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ABSTRACT 


The  Fire  Serviee  is  at  an  important  erossroads  in  its  history.  The  inereasing  threat  of 
terrorism,  along  with  the  need  for  the  Fire  Serviee  to  make  a  major  eontribution  to  the 
Homeland  Seeurity  effort,  eompels  firefighters  to  expand  upon  their  traditional  mission 
requirements.  They  must  deepen  and  widen  the  skills  they  possess  for  responding  to  all 
hazard  ineidents,  and  must  also  eover  the  terroristie  events  and  weapons  of  mass  destruetion 
ineidents  that  are  inevitable  during  this  war  on  terrorism.  This  will  only  be  possible  if  the 
Fire  Serviee  embraees  major  eoneeptual  ehanges  in  the  way  that  it  trains  and  exereises  its 
members.  In  many  respeets,  that  exereise  and  training  system  is  exeellent  and  serves  as  a 
model  for  other  diseiplines.  In  other  ways,  however,  it  tends  to  be  flawed  and  burdensome, 
preventing  the  Fire  Serviee  from  taking  major  strides  forward  in  its  training  efforts  - 
espeeially  with  respeet  to  the  use  of  metries  to  objeetively  evaluate  performanee  eapabilities 
for  Homeland  Seeurity-related  operations,  and  the  embedding  of  assessment  teehniques  in  a 
broader  system  to  provide  for  improved  performanee. 

This  thesis  examines  the  eurrent  approaeh  taken  by  the  Fire  Serviee  to  training 
and  exereising,  espeeially  in  the  Fire  Department  of  New  York  (FDNY),  and  identifies 
gaps  and  problems  in  those  existing  systems. Then,  the  thesis  examines  relevant  “best 
praetiees”  from  the  United  States  Military  and  the  private  seetor  that  might  be  applied  to 
the  shortfalls  in  Fire  Serviee  training  and  exereising.  The  remainder  of  the  thesis  proposes 
ehanges  to  the  training  and  exereise  system,  designed  to  make  a  measurable,  sustained 
impaet  on  the  eapabilities  of  the  fire  serviee  to  aoeomplish  Homeland  Seeurity  and 
traditional  mission  requirements. 
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I:  INTRODUCTION 


A,  PURPOSE 

The  Fire  Service  is  at  an  important  crossroads  in  its  history.  The  increasing  threat 
of  terrorism,  along  with  the  need  for  the  Fire  Service  to  make  a  major  contribution  to  the 
Homeland  Security  effort  compels  firefighters  to  expand  upon  their  traditional  mission 
requirements.  They  must  deepen  and  widen  the  skills  they  possess  for  responding  to  all- 
hazard  incidents,  so  that  they  can  be  ready  to  apply  those  skills  to  the  terroristic  events 
and  weapons  of  mass  destruction  incidents  that  are  inevitable  during  this  war  on 
terrorism.  This  will  only  be  possible  if  the  Fire  Service  embraces  major  conceptual 
changes  in  the  way  that  it  trains  and  exercises  its  members.  In  many  respects,  that 
exercise  and  training  system  is  excellent  and  serves  as  a  model  for  other  disciplines.  In 
other  ways,  however,  it  tends  to  be  flawed  and  burdensome,  preventing  the  Fire  Service 
from  taking  major  strides  forward  in  its  training  efforts  -  especially  with  respect  to  the 
use  of  metrics  to  objectively  evaluate  performance  capabilities  for  Homeland  Security- 
related  operations,  and  the  embedding  of  assessment  techniques  in  a  broader  system  to 
provide  for  improved  performance. 

This  thesis  examines  the  current  approach  taken  by  the  Fire  Service  to  training 
and  exercising,  especially  in  the  Fire  Department  of  New  York  (FDNY),  and  identifies 
gaps  and  problems  in  those  existing  systems.  Then,  the  thesis  examines  relevant  “best 
practices”  from  the  United  States  Military  and  the  private  sector  that  might  be  applied  to 
the  shortfalls  in  Fire  Service  training  and  exercising.  The  remainder  of  the  thesis  proposes 
changes  to  the  training  and  exercise  system,  designed  to  make  a  measurable,  sustained 
impact  on  the  capabilities  of  the  fire  service  to  accomplish  Homeland  Security  and 
traditional  mission  requirements. 


*  An  all-hazards  event  is  defined,  for  the  purposes  of  this  thesis,  as  common  fires  and  emergencies  that  the  first 
responder  community  responds  to  on  a  daily  bases  to  protect  the  life  and  property  of  the  citizens  that  they  protect.  They 
can  range  from  fire  and  medical  emergencies  to  building  collapses,  flooding  and  hurricanes,  anything  that  puts  the 
public  at  large  in  danger. 

2 

A  terrorist  event  is  defined,  for  the  purpose  of  this  thesis,  as  an  event  that  involves  a  CBRNE  agent  or  precursor  that 
was  intentionally  placed  to  do  harm  to  a  person  or  object  with  the  desire  to  influence  change. 
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B,  THESIS  ARGUMENTS 

This  thesis  will  argue  that  while  the  Fire  Serviee  has  pioneered  the  praetiee  of 
training  in  harsh,  realistic  conditions,  and  has  developed  an  excellent  evaluation  process 
to  assess  the  performance  and  attainment  of  minimum  standards  by  new  recruits 
(“Probies”  ),  the  broader  training  and  exercise  program  suffers  from  major  flaws. 
Compared  to  the  U.S.  military  and  the  private  sector,  the  fire  Service  has  made  very  little 
progress  in  the  use  of  metrics  to  assess  performance  in  exercises,  and  to  utilize  those 
performance  metrics  as  part  of  a  sustained  system  to  improve  Fire  Service  capabilities. 
This  is  a  huge  opportunity  for  improvement.  After  having  completed  the  research  for  this 
thesis,  I  am  a  firm  believer  that  “you  can’t  improve  what  you  don’t  measure”  (Harbour, 
1997).  Indeed,  the  ability  of  the  Fire  Service  to  achieve  its  strategic  plan  for  the  future, 
and  be  the  contributor  that  it  must  be  for  Homeland  Security,  is  dependent  upon  the 
development  and  application  of  appropriate  metrics  in  the  exercise  system. 

The  need  for  progress  is  especially  glaring  in  the  realm  of  multi-agency  exercises. 
In  Homeland  Security,  multiple  agencies  (police,  fire,  public  health,  etc)  must  be  able  to 
work  together  in  response  to  a  terrorist  strike.  Exercises  should  be  structured  to 
objectively  assess  their  ability  to  do  so,  within  a  broader  system  that  would  then  provide 
for  the  improvement  of  capabilities  that  the  exercise  indicated  were  inadequate.  Little  of 
the  sort  exists  today.  Too  often,  exercises  have  fallen  victim  to  the  wishes  of  the 
participating  organizations  (and  their  leaders)  to  look  good.  Nobody  wants  to  be 
perceived  as  an  unprepared  or  incompetent  organization.  Hence,  in  many  exercises 
(multi-jurisdictional,  multi-agency  or  local),  serious  assessments  of  performance  never 
happen.  All  the  agencies  hold  a  group  hug,  sing  kum-by-yah,  slap  each  other  on  the  rear 
end  and  go  home,  complacent  in  their  mediocrity.  There  is  no  attempt  at  real  institutional 
change  or  improvement  of  preparedness  levels.  The  Fire  Service  can,  and  must,  do  better 
than  this. 

The  Fire  Service  is  also  faced  with  the  potential  of  becoming  a  major  player  or  a 
minor  participant  in  the  consequence  management  of  hazardous  incidents.  There  are 
entities  in  this  widening  playing  field  of  terrorism  response  that  are  poised  to  profess 
expertise  and  dominance  over  the  operational  world  of  consequence  management,  once 

^  Probie  is  the  name  the  FDNY  calls  a  first  year  firefighter.  It  is  short  for  probationary  firefighter. 
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the  sole  possession  of  the  Fire  Serviee.  It  (the  Fire  Serviee)  must  eonfront  this  ehallenge 
and  re-exert  its  eapability  and  profieieney  that  exists  beeause  of  its  strenuous  training 
ideology  that  it  professes  and  implements.  It  must  prove  its  ability  to  eontinue  to  be  the 
primary  foree,  not  through  rhetorie,  but  through  aetion,  the  aetions  of  superior 
profieieney  in  the  essential  tasks  that  are  needed  to  aoeomplish  their  mission.  They  must 
eontinue  their  efforts  for  operational  and  equipment  eompeteney,  that  is  enhaneed  by 
effeetive  training.  They  must  elevate  their  preparedness  assessment  that  will  result  from 
effieient  produetion  and  measurement  of  operational  exereises.  And  they  must  always 
profess  their  readiness  that  will  result  from  the  aeeumulative  effeets  of  all  the  former. 

The  Fire  Serviee  must  use  this  opportunity  to  revitalize  and  strengthen  the  value 
of  exereises  by  leveraging  federal  grant  funds  to  build  an  exereise  system  that  will 
effeetively  refleet  their  eapability  and  superiority  over  other  organizations  that  eovet  their 
missions.  They  must,  through  this  revitalized  exereise  program,  establish  their 
preparedness  and  solidify  their  rightful  position  as  a  major  eontributor  to  Homeland 
Seeurity.  This  ean  only  be  aeeomplished  if  they  develop  proper  metries  to  measure  their 
aeeomplishments  and  the  aeeomplishments  of  rival  ageneies  so  that  their  expertise  and 
eompetenee  will  show  above  all  pretenders.  In  turn,  eompeteney  will  make  poliey  makers 
take  notiee.  The  war  on  terrorism  and  loeal  preparedness  is  a  joint  effort,  not  the  sole 
domain  of  a  single  ageney. 

The  U.S.  military  and  the  private  seetor  have  developed  better  praetiees  for 
exereises  and  training  that  are  potentially  of  great  value  to  the  Fire  Serviee.  While  the 
Fire  Serviee  has  its  own  unique  requirements  and  eharaeteristies,  and  no  one  ageney  ean 
eopy  the  exereise  system  from  another,  the  researeh  of  the  literature  indieates  that  a 
number  of  key  improvements  ean  be  made  if  the  Fire  Serviee  draws  from  the  best 
praetiees  in  industry  and  the  military.  These  improvements  will  enable  fire  ehiefs  to 
assess  the  eapability,  preparedness  and  readiness  levels  of  the  units  within  their  battalions 
and  their  departments  as  a  whole.  They  will  also  allow  performanee  eomparisons  and 
assessments  between  diseiplines  when  used  in  multi-ageney  exereise  seenarios.  In 
partieular,  this  thesis  proposes  the  adoption  of  a  metrie  system  that  eould  be  applied  to 
the  training  and  exereise  programs  of  first  responders,  speeifieally  the  Fire  Serviee,  to 
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afford  the  leadership  of  these  organizations  the  ability  to  assess  the  preparedness  of  their 
members  to  manage  the  consequence  phase  of  an  all-hazards  or  a  terrorist  event. 


C.  RECOMMENDATIONS  OF  THE  THESIS 

The  recommendations  that  will  be  presented  in  this  thesis  will  result  in  a 
significant  advantage  for  the  Fire  Service  to  predict  and  assess  its  preparedness  over  other 
organizations. 

The  first  recommendation  is  that  FDNY  and  all  other  participating  agencies  in  the 
city-wide  exercise  program  accelerate  the  development  of  an  Exercise  Design  Teams 
(EDT)  as  called  for  by  the  Homeland  Security  Exercise  and  Evaluation  Program 
(HSEEP).  Each  agency  should  also  develop  an  Executive  Committee  for  Preparedness 
Evaluation  (ECPE).  This  would  enhance  the  coordination  and  cooperation  among 
agencies  and  establish  a  single  point  of  contact  and  accountability  within  the  agency. 

The  next  recommendation  would  be  to  design  exercises  that  are  simple  and 
focused  to  specific  goals  and  objectives.  Many  exercises  today  cover  too  many  diffuse 
objectives.  The  EDT  should  restructure  exercises  so  that  they  concentrate  on  the  key 
competencies  and  objectives  of  EDNY’s  strategic  plans.  Scenarios  should  be  realistic, 
and  the  performance  measures  for  each  exercise  should  be  specified  with  much  greater 
clarity.  This  will  also  help  facilitate  comparisons  among  exercises  and  agencies. 

Exercises  that  are  multi-jurisdiction  or  multi-agency  should  stress  the  need  to 
follow  an  appropriate  Incident  Management  System  (IMS),  and  the  agencies  should  be 
held  accountable  for  improving  any  deficiencies  in  performance. 

Accomplishing  these  objectives  will  require  that  the  Eire  Service  answer  the 
“who,  what,  where,  why  and  how”  question.  The  answer  to  the  why  question  (i.e.,  why 
should  we  restructure  the  fire  service’s  approach  to  exercising)  -  is  clear:  building  a 
metrics-driven  exercise  system  is  essential  to  taking  the  already  excellent  capabilities  of 
the  Eire  Service  to  the  next  level,  so  that  the  Service  can  be  a  first-rate  contributor  to 
Homeland  Security  (and  be  even  better  than  it  is  today  in  performing  traditional 
missions).  The  other  questions  are  more  difficult  to  answer,  and  provide  the  primary 
research  focus  of  this  thesis.  Who  exactly  should  be  exercised  (ranging  from  performance 
assessments  of  individuals  through  units,  battalions  and  divisions)?  What  skills  should  be 
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measured?  When  and  how  ean  those  assessments  be  made  in  an  objeetive  way,  as 
opposed  to  the  eurrent  relianee  on  subjeetive  eriteria  (or  no  real  criteria  at  all)  prevalent 
in  too  many  exercises  today? 

It  is  further  recommended  that  objective  performance  criteria  be  developed  for 
each  of  the  critical  tasks  that  firefighters  must  accomplish  within  a  given  mission. 
Methodologies  for  developing  these  metrics  can  be  derived  from  a  number  of  models  in 
industry  and  the  military,  and  from  the  sizeable  academic  literature  which  has  emerged 
on  the  topic  (which  is  reviewed  and  critiqued  in  chapter  3).  The  selection  and  application 
of  those  metrics  should  reflect  the  recognition  that  more  criteria  are  not  necessarily  better. 
The  Fire  Service  should  measure  only  the  aspects  of  an  exercise  that  is  pertinent  to  the 
goals  and  objectives  of  that  particular  exercise  and  scenario.  These  goals  and  objectives 
should  reflect  the  broader  strategic  plan  of  the  organization,  rather  than  a  vague  sense  of 
what  is  desirable. 

In  applying  these  metrics  (the  “when”  question),  the  Fire  Service  should  adopt  a 
“before,  during  and  after”  approach.  That  is,  the  service  should  1)  assess  the  capabilities 
of  an  individual  or  unit  before  training  or  exercising;  2)  assess  performance  during  the 
exercise;  and  3)  measure  the  performance  after  the  exercises  (and  after  any  remedial 
training  programs)  are  concluded  so  that  commanders  have  an  accurate  assessment  of 
overall  capabilities. 

As  in  the  military,  comprehensive  After  Action  Reviews  (AAR)  must  become 
standard  components  for  exercises,  so  that  good  practices  can  be  reinforced  while 
deficiencies  are  identified  and  corrected. 

Most  important,  the  desire  of  stakeholders  to  make  their  organizations  look  good 
in  exercises  cannot  be  allowed  to  corrupt  the  exercise  process,  or  weaken  the  value  of 
exercises  to  identify  performance  gaps  and  drive  performance  improvements. 

The  military  commander  is  responsible  to  assess  the  capabilities  of  his  command 
so  that  preparedness  and  readiness  can  be  predicted.  To  help  make  improvements  in  this 
regard,  it  is  recommended  that  the  Fire  Service  borrow  from  this  military  “best  practice” 
in  how  assessments  are  reported  and  used  within  the  organizations.  The  military  often 
involves  very  senior  levels  of  the  organization  in  the  training  process,  by  training  one 
level  down  and  evaluating  two  levels  up.  The  Fire  Service  hierarchy  is  generally  far 


5 


more  distant  from  equivalent  training  activities,  weakening  their  ability  to  use  the  system 
as  an  effective  tool  to  understand  and  improve  service  capabilities.  Adopting  a  system  for 
closer  leadership  involvement  would  be  of  significant  benefit. 


D,  SCOPE  OF  RESEARCH 

This  thesis  includes  the  review  of  the  current  exercise  system  that  is  being  used 
throughout  the  first  responder  community  and  is  supported  by  the  Department  of 
Homeland  Security,  Office  of  Domestic  Preparedness"^.  This  review  is  done  in  order  to 
show  the  types  of  exercises  in  use  today  to  establish  a  baseline  to  discuss  improvements 
that  can  be  accomplished. 

This  thesis  will  review  the  United  States  Army  Training  Manual,  Training  the 
Force^,  and  focus  on  their  assessment  strategy  to  evaluate  performance  and  assess  combat 
readiness.  This  will  ground  our  recommendations  in  a  proven  practice  of  evaluation, 
feedback,  and  assessment.  Though,  not  perfect,  the  military  system  has  valuable  concepts 
to  offer  this  discussion. 

This  thesis  will  acquaint  the  reader  with  various  performance  evaluation  models 
that  are  in  use  in  the  industrial  sector  to  evaluate  a  corporation  and  its  performance  on 
many  different  levels  from  personnel  and  production  processes  to  financial  stability  and 
provide  a  comparison  chart  for  eight  of  the  most  popular  models.  This  will  enable  the 
Fire  Service  to  adopt  proven  evaluation  practices  that  will  result  in  heightened 
preparedness  and  readiness. 

Also  this  thesis  will  analyze  the  training  program  of  the  FDNY  and  discuss  some 
of  the  challenges  that  it  faces  now  and  in  the  future.  This  thesis  will  also  outline  criteria 
for  measuring  performance  that  should  guide  the  creation  of  metrics  to  validate 
preparedness.  Finally,  this  thesis  will  propose  a  metric  system  that  will  allow  the 


The  Department  of  Homeland  Seeurity,  Offiee  for  Domestie  Preparedness,  has  published,  Homeland 
Seeurity  Exereise  and  Evaluation  Program,  whieh  serves  as  the  National  Exereise  Program  guidanee 
doeument. 

^  This  doeument,  the  United  States  Army’s,  Training  Manual,  Training  the  Foree,  serves  as  the  overarehing 
training  manual  for  all  the  military  serviees  and  treats  training  and  operations  as  a  eombined  effort  of  all 
entities  to  eomplete  the  task.  This  serves  as  a  great  example  for  a  multi-ageney,  multi-jurisdietional 
eombined  effort  at  terroristie  aetivity.  No  one  ageney  is  the  end  all  and  be  all  for  erisis  and  eonsequenee 
operations. 
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leadership  of  the  organization  make  an  aecurate  assessment  of  the  capability  of  the  first 
responder  standing  force. 


E.  RESEARCH  METHODOLOGY  AND  THESIS  STRUCTURE 

The  research  methodology  begins  with  a  comprehensive  review  of  literature 
relating  to  performance  measurement,  strategic  planning,  evaluations  and  metric 
methodologies.  The  review  of  literature  included  books,  professional  journals. 
Department  of  Defense  publications.  Department  of  Homeland  Security  publications. 
Congressional  legislation.  Fire  Department  of  New  York  City  policies  and  procedures, 
and  private  internet  sources.  The  best  practices  of  the  organizations  involved  in  the 
literature  will  be  extracted  and  used  to  recommend  a  metric  system  to  evaluate  and  assess 
the  preparedness  of  first  responder  units  and  organizations. 

The  thesis  begins  by  reviewing  the  current  exercise  system  of  the  Fire  Service  in 
Chapter  II  (and,  more  specifically,  FDNY),  and  assessing  how  that  system  is  evolving  in 
response  to  the  new  requirements  driven  by  Homeland  Security.  Chapter  II  also  analyzes 
the  implications  of  emerging  Federal  requirements  on  the  exercise  system  for  FDNY 
efforts  to  improve  its  exercise  system.  The  analysis  identifies  a  number  of  excellent 
practices  in  the  Fire  Service  that  need  to  be  extended  throughout  its  exercise  system,  and 
also  significant  new  opportunities  to  build  metrics  into  FDNY’s  exercise  evaluation 
system.  Chapter  III  examines  best  practices  from  the  U.S.  military  and  from  the  private 
sector  that  might  be  applied  to  the  Fire  Service,  and  reviews  the  academic  literature  on 
methodologies  to  build  metrics  and  performance  measurement  systems.  Chapter  IV 
applies  lessons  learned  from  the  review  of  the  literature  and  best  practices  to  the 
development  of  performance  criteria  for  use  by  the  Fire  Service.  Chapter  V  proposes  how 
those  metrics  might  be  incorporated  into  a  broader  restructuring  of  the  Fire  Service’s 
exercise  system,  with  Chapter  VI  offering  my  conclusions. 
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II:  THE  CURRENT  EXERCISE  SYSTEM  OF  THE  FIRE  SERVICE: 

EXCELLENCE  AND  OPPORTUNITIES  FOR  IMPROVEMENT 

A,  INTRODUCTION 

As  first  responders  to  fires,  publie  safety  and  medieal  emergeneies, 
disasters  and  terrorist  aets,  the  FDNY  proteets  the  lives  and  property  of 
New  York  residents  and  visitors.  The  Department  advanees  publie  safety 
through  its  fire  prevention,  investigation  and  education  programs.  The 
timely  delivery  of  these  services  enables  the  FDNY  to  make  significant 
contributions  to  the  safety  of  New  York  City  and  homeland  security 
efforts. 

Core  values:  Service,  Bravery,  Safety,  Honor  Dedication,  and 
Preparedness. 

-The  Mission  Statement  of  the  Fire  Department  of  New  York  as 
stated  in  their  Strategic  Plan  of  2004-20,  dated  January  1,  2004 

The  realities  of  the  post  9/11  world  have  sown  the  seeds  of  change  in  the  Fire 
Service.  The  Fire  Department  of  New  York  City  (FDNY)  has  begun  this  process  of 
change  by  redefining  its  mission  statement,  restating  its  core  values  and  publishing  its 
first-ever  strategic  plan  to  guide  the  Department  in  its  effort  to  meet  its  new 
responsibilities  in  the  war  on  terrorism.  The  Department  is  not  only  following  this  new 
strategic  plan,  but  more  importantly,  the  leadership  and  the  members  are  now  thinking 
strategically  about  how  to  build  its  capabilities  to  meet  the  requirements  of  Homeland 
Security. 

Some  features  of  the  current  exercise  system  are  well  suited  to  strengthening  Fire 
Service  capabilities  (as  will  be  demonstrated  later  in  this  chapter).  Overall,  however,  the 
Fire  Service  can  not  validate  its  preparedness  for  terrorism  response  because  it  uses  a 
system  that  is  based  in  observational  evaluation  of  capabilities,  which  tends  to  be 
subjective  and  potentially  biased.  The  fire  service  has  long  believed  that  training  is  a 
continuous,  career  long  endeavor  that,  if  accomplished  correctly,  will  produce  competent, 
confident,  disciplined  and  adaptive  firefighters  and  officers.  It  was  an  easy  methodology 
for  the  fire  service  to  follow  in  the  past.  We  could  train  firefighters  cheaply  and 
efficiently  through  on  the  job  experience  with  a  heavy  workload  and  plenty  of 
experienced  veterans  willing  to  share  their  accumulated  knowledge.  In  the  post  9/1 1 
world,  this  is  no  longer  the  case.  The  Fire  Service  has  entered  into  a  new  era  of 
requirements  for  performance  standards,  competencies,  evaluations  and  assessments. 
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The  Fire  Serviee  has  embraced  and  adopted  the  “National  Exercise  Program”^ 
(NEP)  as  part  of  their  efforts  to  evaluate  the  performance  and  competency  of  its 
members.  They  are  now  faced  with  the  opportunity  to  restructure  their  whole  approach  to 
training  that  will  enable  them  to  become  an  effective  contributor  in  the  war  against 
terrorism.  This  can  happen  by  using  the  exercise  as  a  final  evaluation  and  assessment  tool 
to  validate  the  training  program.  The  adoption  of  metrics  within  the  exercise  system  and 
the  whole  training  program  will  enable  the  Eire  Service  to  measure  their  performance, 
identify  gaps  within  the  training  system,  test  new  procedures  and  plans  for  effectiveness, 
and  provide  an  assessment  of  the  organization’s  capability.  Metrics  will  enable  the  Eire 
Service  to  establish  a  foundation  upon  which  knowledge  gained  through  experience  can 
be  transferred  to  new  learners  through  the  learning  cycle,  and  thereby  reinforce  a 
sustained  effort  to  build  the  new  capabilities  at  the  unit  level. 

The  fire  service  is  not  alone  in  facing  this  challenge.  The  technological  age  (the 
late  1900’s  into  the  2E*  century)  has  brought  with  it  an  explosion  of  knowledge  that  feeds 
a  technological  boom  world-wide.  We  no  sooner  master  a  new  technology  than  it  is 
replaced  by  a  more  advanced  one  that  requires  additional  skills  and  knowledge. 
Performance  and  performance  measurement  becomes  the  underlying  indicator  that  an 
institution  has  successfully  achieved  knowledge  continuity  and  growth.  Many  researchers 
have  studied  these  phenomena  and  examined  the  ways  that  we  learn  and  transfer 
knowledge.  The  one  thing  they  all  agree  upon  is  that  in  order  to  know  where  we  must  go 
(that  is,  how  performance  needs  to  be  increased),  we  must  remember  where  we  came 
from  (our  past  performance  level)  and  where  we  are  at  the  present  time  so  that  a 
systematic  way  of  improving  performance  can  be  structured.  Each  step  along  the  way 
should  require  a  metric  or  set  of  metrics  to  guide  us  along  the  path.  Moreover,  every 
organization  should  have  a  strategic  plan  with  goals  and  objectives  that  are  clearly 
discernable,  with  performance  metrics  embedded  in  an  overall  exercise  and  training 
system  that  can  support  the  implementation  and  achievement  of  that  plan.  The  training 
that  an  organization  develops  must  lead  to  the  achievement  of  the  organizations 
objectives  and  goals.  The  only  way  that  an  organization  will  know  if  it  has  met  its  goals 

®  The  National  Exercise  Program  (NEP)  has  become  the  national  standard  to  which  exercises  are  compared.  The 
program  was  created  by  the  Office  for  Domestic  Preparedness  and  has  established  the  HSEEP,  Homeland  Security 
Exercise  and  Evaluation  Program  document  as  the  cornerstone  of  the  program. 
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and  adhered  to  its  strategic  plan  is  through  the  metrics  that  it  creates  to  measure  its  past, 
present  and  future  capabilities. 

This  Chapter  will  examine  how  effectively  the  fire  service  is  meeting  these 
overall  requirements  for  effective  performance  measurements  and  drill  down  into  some 
specific  opportunities  for  improvement.  The  chapter  begins  by  reviewing  some  key 
sources  of  Federal  guidance  in  restructuring  exercises  for  homeland  security,  and 
summarizes  the  key  features  of  the  existing  training  and  exercise  system.  This  chapter 
then  examines  emerging  challenges  that  the  system  confronts,  especially  those  related  to 
terrorism  response.  After  reviewing  some  important  new  initiatives  designed  to  meet 
these  challenges,  the  chapter  concludes  by  highlighting  some  remaining  requirements  for 
change. 


B,  THE  HOMELAND  SECURITY  EXERCISE  AND  EVALUATION 
PROGRAM  (HSEEP) 

1,  Overview  of  the  HSEEP 

Past  experience  has  shown  that  preparedness  efforts  are  key  to  providing 
an  effective  response  to  major  terrorist  incidents  and  natural  disasters. 
Therefore,  we  need  a  comprehensive  national  system  to  bring  together  and 
command  all  necessary  response  assets  quickly  and  effectively.  We  must 
equip,  train  and  exercise  many  different  response  units  to  mobilize  for  any 
emergency  without  warning. 

-  National  Strategy  for  Flomeland  Security,  July  2002 

The  Department  of  Homeland  Security,  Office  for  Domestic  Preparedness 
(ODP)^,  has  implemented  a  program  of  financial  and  direct  support  designed  to  assist 
state  and  local  governments  with  the  development  and  implementation  of  an  exercise  and 
evaluation  program  to  assess  and  enhance  domestic  preparedness.  This  program  provides 
the  blueprint  for  the  Fire  Service  to  produce  and  perform  an  exercise  and  provides 
assessment  tools  to  gauge  preparedness.  The  program  proposes  a  cycle  of  exercises  of 
increasing  complexity,  two  tabletops  and  an  operational  exercise  that  are  designed  to  test 
performance  as  well  as  the  adequacy  of  plans,  policies  and  procedures.  The  scenarios  that 
the  program  suggests  are  threat-based,  realistic,  and  generally  involve  multi-agencies  and 


’’  U.S.  Department  of  Homeland  Seeurity,  Office  for  Domestic  Preparedness,  Homeland  Security  Exercise 
and  Evaluation  Program  (HSEEP), is  available  at  www.oip.usdoi.gov/odp.  September  2003. 
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jurisdictions.  All  the  exercises  are  evaluated  and  the  host  organizations  are  provided  with 
After  Action  Reports  and  Corrective  Action  Plans.  Lessons  learned  and  best  practices 
from  the  exercises  are  recorded  and  shared  with  the  homeland  security  community. 

The  evaluation  plan  for  the  program  is  broken  down  into  two  parts,  evaluation  of 
tabletop  exercises  and  evaluation  of  operational  field-type  exercises.  The  Tabletop 
evaluations  consist  of  the  activity  and  discussions  that  goes  on  during  the  exercise  which 
is  recorded,  analyzed  and  formatted  in  a  summary  report. 

The  evaluation  of  the  operational  field-type  exercise  consists  of  an  analysis  of 
individual  performance,  team/discipline/department-level  performance,  and  a 
community/mission-level  performance.  The  performances  are  observed  and  recorded,  and 
the  players  are  given  an  opportunity  during  a  hot-wash  to  discuss  the  actions  and 
performance  of  their  respective  mission  areas.  The  After  Action  Report  (AAR)  is  the 
crucial  document  that  documents  performance  and  captures  the  analysis  of  the  observers. 
It  provides  a  description  of  events,  issues  that  arose  and  how  they  were  handled,  and 
recommendations  for  corrective  actions.  The  final  piece  to  the  evaluation  system  is  the 
Corrective  Action  Plan  (CAP)  that  recommends  the  changes  that  need  to  be  made  to  the 
plans,  policies  and  procedures  of  the  organizations. 

The  HSEEP  outlines  an  evaluation  process  that  uses  objective  observers  to 
record  the  performance  of  the  players  and  the  issues  that  may  arise  during  play.  Many 
times  the  observers  are  from  the  very  organizations  participating  in  the  exercise  which 
may  lead  to  biases  arising  in  the  final  analysis  of  the  AAR  and  the  CAP.  This  is  a  major 
problem  that  needs  to  be  remedied.  Organizations  cannot  be  relied  upon  to  evaluate 
themselves  in  the  absence  of  genuine,  objective  metrics  and  a  system  to  minimize  the 
inevitable  desire  of  organizations  “to  make  themselves  look  good.”  If  an  exercise  is 
going  to  be  used  as  an  assessment  tool  for  a  department  or  a  municipality,  then  it  must  be 
independent,  no-notice,  fully  evaluated  and  measured  through  objective  criteria.  The 
results  must  also  be  verified  multiple  times  over  a  period  of  time  that  is  appropriate  for 
the  organization.  Moreover,  while  the  AAR  and  the  CAP  can  be  transformed  into  good 
tools  for  the  Eire  Service  to  adopt  into  their  programs,  the  exercises  must  be  tied  back  to 
training  organizations  to  complete  the  training  cycle.  The  HSEEP  falls  short  in  specifying 
how  to  transfer  knowledge  and  learning  back  to  training,  so  that  performance  shortfalls 
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can  be  remedied  in  a  struetured,  eonsistent  way.  In  sum:  the  HSEEP  is  valuable,  but  the 
Eire  Serviee  needs  to  move  beyond  it  in  building  a  metries-driven  exereise  system. 


C.  CURRENT  FDNY  TRAINING  AND  EXERCISE  MODELS 

The  EDNY  uses  exereises  as  a  means  to  aeeomplish  hands-on  training,  to 
reinforee  training  objeetives  and  goals,  and  as  a  means  to  validate  training  eourses. 
Exereises  are  a  standard  at  the  end  of  firematie  and  hazmat  eourses  to  put  learned  skills  to 
the  test  and  to  integrate  the  many  skills  required  to  eomplete  a  mission.  Most  of  these 
exereises  are  realistie  scenario-based  exereises,  and  are  observed  and  evaluated  by  the 
instructor  to  correct  any  deficiencies  and  to  summarize  the  learning.  A  well  designed  and 
executed  exereise  is  also  an  effeetive  means  to: 

•  Test  polieies,  plans,  and  proeedures 

•  Clarity  and  train  personnel  in  roles  and  responsibilities 

•  Identify  gaps  in  resourees 

•  Improve  individual  performanee 

•  Identify  opportunities  for  improvement 

•  Improve  interagency  coordination  and  communications 

The  EDNY  is  striving  to  improve  its  exereise  model  to  mirror  the  elements 
present  in  the  HSEEP  model  and  has  seeured  funding  to  establish  an  Exereise  Design 
Team.  This  team  will  be  able  to  provide  assistanee  to  Battalion  and  Division  eommanders 
who  wish  to  produee  an  exereise  for  their  eommand  so  that  they  ean  evaluate  eapabilities, 
preparedness  and  readiness  to  aeeomplish  speeifie  missions  within  their  jurisdietions. 

This  Exercise  Design  Team  (EDT)  emulates  the  proposed  team  contained  within 
the  HSEEP  but,  as  opposed  to  a  transitory  team,  it  would  be  a  permanent  unit  that  will  act 
as  a  liaison  for  the  small,  fire  department  exereises  as  well  as  multi-agency,  multi- 
jurisdictional  exercises.  The  EDT  will  standardize  the  exereise  proeess  and  pull  the  best 
praetiees  and  evaluations  from  the  NEP,  military  and  industrial  models. 

In  terms  of  training,  some  of  the  very  best  praetiees  in  the  Eire  Serviee  are 
reflected  in  the  Probationary  Eirefighter  Sehool  Model  -  praetiees  that  might  be  applied 
much  more  widely  by  the  EDT  to  other  exereise  and  training  programs  in  the  serviee. 
The  motto  that  guides  the  training  attitude  of  the  EDNY  is  emblazoned  above  the 
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entrance  to  the  Probie  equipment  room:  “Let  no  man’s  ghost  return  to  say  his  training  let 

o 

him  down”.  The  standards  that  are  set  by  this  one  motto  are  high,  and  instructors  push 
the  limit  to  enable  Probies  to  meet  that  standard.  The  knowledge  that  is  learned,  the  skills 
that  are  honed  and  the  competencies  that  are  achieved  start  the  new  firefighters  on  their 
way  in  an  effective  fashion.  It  is  the  combination  of  what  is  learned  and  measured  in  the 
initial  stages  at  the  academy  that  will  stay  with  them  their  whole  career.^  The 
Probationary  Firefighter  School  (PFS)  uses  a  semi-military  disciplinary  style  that  creates 
a  structured  regiment  of  learning.  The  probie,  as  they  are  called,  is  introduced  to  fire 
tactics,  fire  prevention,  emergency  medical  procedures  and  hazardous  materials 
operations  over  a  twelve  week  period. 

Academically,  performance  is  measured  and  tracked  using  a  series  of  weekly 
quizzes,  a  midterm  and  final  exam,  and  they  are  also  tightly  monitored  over  their  twelve 
week  period  at  the  school.  The  academic  performance  is  closely  watched  and  the  probie 
is  given  every  opportunity  to  improve  through  individual  training,  peer  training  or  extra 
sessions  if  the  need  arises. 

The  hands-on  training  (or  “train  as  you  fight”), accounts  for  60%  of  the  training 
skills  that  are  taught  and  measured.  This  is  evaluated  by  their  instructor,  either  by 
observation  that  a  skill  is  performed  or  with  reference  to  a  checklist  of  steps  in  a  properly 
performed  evolution,  forty-one  times  over  a  twelve  week  period.  If  any  evaluation  is 
unsatisfactory  or  the  probie  fails  to  perform  adequately,  the  problem  is  addressed  forth¬ 
with  using  an  intimidation  tactic  that  places  the  probie  in  the  office  of  the  Captain-in- 
Charge  of  the  school  for  a  ‘very  personal’  interview. 

This  model  works  well  for  many  reasons.  It  highlights  the  seriousness  of  the 
career  that  the  probie  is  about  to  enter.  It  strives  to  prepare  them  for  the  life  or  death 
situations  that  they  will  find  themselves.  Every  tactic  and  procedure  in  the  twelve  weeks 
of  training  prepares  them  for  the  reality  of  firefighting.  They  train  as  they  fight,  with  the 
same  equipment,  in  realistic  environments,  and  with  the  same  high  stress.  Every 
instructor  strives  to  place  the  probie  inside  the  organized  chaos  that  is  the  fire  floor.  The 


®  Unknown  author 

^  This  information  comes  from  an  interview  with  Lt.  John  Regan,  the  former  Executive  Officer  of  the  Probationary 
Firefighting  School,  NYC  Fire  Academy,  Randall’s  Island,  NY,  10035,  and  the  author,  April  29,  2004. 

Train  as  you  fight  is  the  motto  of  the  military  training  manual. 
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training  and  evaluation  process  instills  a  sense  of  discipline  and  respect  for  chain  of 
command  that  exists  in  the  fire  department.  And  finally,  the  PFS  model  instills  within  the 
probies  the  ability  and  the  necessity  to  take  an  order  without  question. 

The  final  evaluation  of  all  the  new  probies  comes  from  the  Chief-in-Charge  of 
Training  who  has  been  actively  evaluating  and  observing  the  formation  of  these  skill 
levels.  All  the  evaluation  reports  are  at  his  disposal  to  formulate  his  assessment  of  their 
preparedness  to  join  the  active  force.  If  a  probie  fails,  they  will  have  to  repeat  the  twelve 
weeks  or  be  terminated  from  the  force.  This  is  a  tough  decision  but  one  that  has  to  be 
enforced  for  the  good  of  the  service 

Other  models  (and  potential  best  practices  on  which  to  draw  more  widely)  exist  in 
specialized  training  programs.  The  CFRD'^  (Certified  First  Responder,  Defibrillator) 
Certification  Model  is  valuable  in  this  regard.  This  training  in  medical  emergency 
procedures  is  strictly  monitored  by  the  Fire  Department  Medical  Affairs  Division  and  is 
evaluated  on  a  regular  basis.  The  students  are  tested  academically  through  quizzes  and 
examinations  and  must  pass  a  New  York  State  mandated  exam  before  being  certified. 
There  is  also  a  hands-on  skills  workshop  that  is  observed  and  recorded  on  a  check-off  list 
basis.  The  evaluation  is  quantified  to  produce  a  performance  profile  on  ability.  The 
course  instills  an  environment  of  seriousness,  capability  evaluation,  accountability,  and 
leadership  involvement.  This  effort  produces  qualified  CFRD  firefighters  that  are  capable 
and  confident  in  their  approach  to  their  work. 

Hazardous  Materials  Training,  both  Operations  and  Technician  Levels,  provide 
another  model  of  how  objective  criteria  can  be  brought  to  bear  in  performance 
evaluations.  This  was  developed  and  is  sponsored  by  the  International  Association  of 
Firefighters  (lAFF)  through  a  grant  process  from  federal  sources.  The  program  strives  to 
deliver  a  course  that  meets  the  highest  standards  for  efficiency  and  effectiveness  for  the 
audience  they  are  addressing  (namely,  the  adult  first  responder).  A  rigorous  Quality 
Assurance  Program  is  employed  to  generate  a  report  containing  selected  performance 

*’  CFRD  or  Certified  First  Responder,  Defibrillator,  is  a  New  York  State  category  especially  created  for 
first  responders  in  NYC  to  fill  the  need  to  have  certified  personnel  placed  quickly  on  the  scene  of  a  cardiac 
emergency  or  other  trauma.  They  yield  patient  care  to  Paramedics  or  EMT’s  upon  their  arrival.  CFRD  is  a 
level  below  an  EMT  (Emergency  Medical  Technician)  who  works  on  a  fire  apparatus  instead  of  an 
ambulance. 

The  Code  of  Federal  Regulations,  OSHA,  CFR  1910.120,  describes  four  levels  of  training  for  Hazardous  Materials 
workers,  Awareness,  Operations,  Technician,  and  Incident  Commander. 
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indicators  that  are  coupled  with  pre-  and  post-  test  scores  that  are  used  to  represent  a 
percent  knowledge  gained.  A  third  test  is  given  to  selected  students  one  year  after  the 
course  to  measure  the  amount  of  knowledge  retained.  This  program  of  Quality 
Assurance,  though  not  new  to  other  disciplines,  is  an  innovative  approach  in  the  Fire 
Service  that  has  proven  success  over  the  years. 

Numerous  other  approaches  to  training  are  used  at  the  Fire  Academy  because  of 
the  cross  section  of  experience  and  seniority  among  the  members  trained.  The  most 
popular  model  mirrors  some  of  the  best  practices  of  the  military  evaluation  program  that 
uses  instantaneous  feedback  and  observation  as  their  assessment  tools.  The  Probationary 
Firefighter  School  uses  this  model  and  expands  upon  it  with  great  success.  Most  other 
courses  used  for  in  service  training  of  new  procedures  (or  little  used  competencies  which 
train  more  senior  personnel)  use  observation  of  hands-on  evolutions  as  their  evaluation 
methods,  with  a  training  mark  given  at  the  end  of  the  session.  End  of  course  reaction 
questionnaires  or  “reactionaires”  (as  they  have  come  to  be  called)  are  used  in  a  limited 
fashion  to  evaluate  instructors,  course  content  and  course  environment. 

D,  CHALLENGES  FACED  BY  THE  FDNY  AND  THE  FIRE  SERVICE 

There  are  many  challenges  that  exist  for  the  Fire  Service  and  the  FDNY  in  their 
training  routines.  The  main  challenge  exists  in  the  department’s  ability  to  take 
capabilities  built  from  the  training  process  and  translate  them  into  an  accurate 
preparedness  model.  The  Fire  Service  needs  to  adhere  to  the  standards^"^  of  OSHA 
1910.120  and  NFPA  472  as  a  minimum  base  line  for  members  operating  in  hazardous 
environments  and  at  terror  events,  build  upon  that  foundation  with  new  standards  that 
will  raise  the  bar  of  preparedness. 

The  largest,  single  challenge  that  faces  the  FDNY  and  the  Fire  Service  is  the 
maintenance  of  competency  in  skill  sets  that  are  seldom  used  but  are  critical  to  complete 
the  missions  for  which  they  were  designed.  Members  remain  competent  in  many  of  the 
skills  used  in  firefighting  and  medical  care  because  they  use  those  skills  daily  in  their 


Reactionaires  is  a  coined  phrase  for  questionnaire  by  the  IBS  in  their  study  by  Tamkin,  et,  ah,  p  3 1 . 

OSHA  1910.120  and  NFPA  472  are  the  standards  currently  recognized  as  the  guiding  influence  when  training 
hazmat  personnel  to  deal  with  hazardous  material  incidents.  They  deal  with  operational  principles  and  personal 
protective  equipment  usage. 
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responses  whieh  are  reinforeed  in  their  experienee  gain.  The  multitudes  of  other  skill  sets 
that  are  used  in  eonjunetion  with  these  often  used  skills  are  praetieed  in  the  firehouse  so 
that  they  are  at  a  high  readiness  posture. 

The  war  on  terrorism  has  mandated  that  the  Fire  Serviee  learn  an  additional  set  of 
skills  that  are  not  readily  experieneed  in  the  field.  We  ean  only  train  in  a  simulated 
environment.  This  presents  a  large  ehallenge  to  the  training  professional  in  the  Fire 
Serviee  to  keep  the  standards  high  while  maintaining  the  interest  and  determination  of  the 
adult,  experieneed,  senior  firefighter. 

The  influx  of  numerous  deteetion  deviees  speeifie  for  terroristie  aetivity,  and  the 
additional  skill  sets  needed  to  evaluate  these  deviees,  has  ereated  another  large  problem 
in  a  foree  that  gains  knowledge  through  experienee  (even  if  simulated).  The  experienees 
are  simply  not  there.  New  learning  and  experienee  teehniques  need  to  be  developed  to 
artifieially  produee  skill  sets  that  are  adaptive  to  the  new  required  skills.  These  skills 
must  also  be  measured  and  reeorded  to  improve  our  posture. 

The  training  model  of  the  FDNY  is  typieal  of  the  Fire  Serviee  in  general  and 
needs  to  inerease  its  flexibility  and  its  dynamies.  The  mission  that  the  Fire  Serviee  has 
undertaken  is  different  from  the  past,  and  requires  new  ideas  and  modern  teehnology  to 
meet  the  rising  ehallenges.  The  leadership  must  be  proaetive  in  its  thinking  and  aetions. 
Evaluation  of  training  in  order  to  evolve  is  tantamount  to  an  effeetive,  eapable,  prepared 
and  ready  foree. 

Single-ageney  exereises  are  the  eapping  stone  that  should  aeeompany  any 
meaningful  training  regiment.  They  validate  the  training  proeess  and  if  measured 
eorreetly,  will  give  the  eommander  a  pieture  of  the  eapability,  preparedness  and  readiness 
of  the  eommand.  The  biggest  ehallenge  of  the  Fire  Serviee  is  to  provide  a  realistie, 
meaningful  exereise  that  will  ehallenge  the  training  of  all  of  its  members  and  supply  the 
proper  measures  to  analyze  the  outeomes. 

Presently,  exereises  run  within  the  department,  for  the  department,  are  speeifie 
training  exereises  that  use  observation  by  the  eommanders  to  evaluate  the  outputs.  They 
are  generally  sueeessful  efforts  partieipated  in  by  motivated  personnel  that  genuinely 
produee  effeetive  results  in  knowledge  transfer  from  members  to  members,  eommanders 
to  members  and  even  members  to  eommanders.  At  the  eonelusion  of  the  exereise. 
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whether  it  is  a  small  end  of  course  evaluation,  hands-on  exercise  or  a  large  operational 
Hi-Rise  exercise,  the  favorite  evaluational  tool  is  observation-based  critique  along  with 
an  After  Action  Discussion  forum. 

Inter-agency  exercises  are  becoming  a  larger  challenge  for  the  department, 
especially  with  regard  to  terrorism  response.  Here,  operational  lines  are  blurred  and 
mission  areas  are  not  defined.  Measurable  metrics  are  frowned  upon  because  they  may 
point  to  a  deficiency  in  cooperation  (which  would  be  politically  embarrassing).  The 
Incident  Commander,  who  may  have  a  relatively  accurate  idea  of  the  capabilities  and 
preparedness  of  his  members,  has  no  effective  assessment  of  the  capabilities  of  other 
agencies.  In  the  past,  this  has  led  to  dangerous  situations  for  the  operational  forces  at 
actual  incidents.  Many  gaps  in  procedures  from  one  organization  to  the  next  become 
apparent,  but  neither  organization  is  willing  to  admit  to  its  own  shortcomings.  Unlike  the 
system  called  for  in  National  Incident  Management  System,  there  is  no  single  manual  that 
describes  procedures  for  joint  operations  that  is  accepted  by  all  players  in  NYC. 

Herein  lies  the  challenge  for  the  FDNY  and  the  City.  The  fact  is  that  at  any  future 
terrorist  event  that  includes  a  consequence  management  response  (traditionally  a 
Fire/EMS  responsibility),  that  response  will  also  include  the  response  of  other  agencies 
that  will  cause  operational  competition,  a  competition  on  a  Command  level  as  well  as  at 
an  operational  level  that  will  lead  to  command  and  control  problems. 

E.  EMERGING  BEST  PRACTICES 

The  FDNY  is  headed  in  the  right  direction  in  establishing  the  Exercise  Design 
Team  as  an  institutional  means  of  coordinating  and  standardizing  exercises  without 
dictating  content  in  an  overly  detailed  way.  In  many  other  respects  as  well,  initiatives  are 
underway  in  EDNY  that  -  as  part  of  a  broader  restructuring  of  the  exercise  system — 
should  be  incorporated  in  that  redesign  process.  Some  of  the  best  practices  are  as  follows: 

•  The  local  division  and  Battalion  Chiefs  have  the  ability  to  produce  training 
exercises  specific  to  their  area. 

•  Chief  Officers  often  take  significant  roles  in  the  production  of  these  single 
agency  exercises. 

•  Policies  and  procedures  are  put  to  the  test  and  evaluated. 
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•  At  the  Fire  Academy,  members  are  trained  in  regularly-scheduled,  hands-on 
exercises  that  strive  to  reinforce  training  across  the  service. 

•  Exercises  typically  use  realistic  scenarios  and  equipment. 

A  new  best  practice  is  emerging  with  regard  to  the  special  challenges  of  the  multi¬ 
agency  operations  and  exercises.  This  has  been  termed  the  Phased  Response  Strategy 
Exercise'^  and  is  being  developed  by  the  EDNY  E*  Division,  in  lower  Manhattan.  It  is 
based  on  the  assumption  that  every  incident  has  a  logical  flow  that  can  be  put  into  phases: 
the  incipient  or  response  phase,  the  mitigation  phase  and  the  overhaul  and  investigative 
phase.  In  the  field,  this  is  a  pattern  that  real  incidents  follow  and  because  of  reality  and 
the  actual  response  patterns  of  the  responding  disciplines  inter-agency  problems  seem  to 
be  non-existent  or  just  melt  away.  Exercises,  in  contrast,  are  different  due  to  the  sterilized 
environment  in  which  the  scenario  occurs.  Every  response  organization  is  unrealistically 
placed  at  the  scene,  if  not  immediately,  then  close  to  it,  where  all  their  resources  and 
assets  can  be  placed  in  play,  which  leads  to  disruptions  and  competition. 

In  the  Phased  Response  Strategy,  agencies  work  on  independent  problems  in 
phases  as  they  would  during  a  real  incident,  each  agency  completing  their  assigned 
missions.  Eor  example,  a  bomb  is  found  in  Penn  Station,  the  major  organizations  that 
would  respond  in  phase  one  to  assess  the  incident  are:  EDNY,  NYPD,  FBI,  and  DEP. 
They  would  logically  and  in  reality  not  respond  at  the  same  time.  Initial  units  on  the 
scene  would  be  EDNY  and  NYPD.  Since  it  is  a  bomb  (phase  two  begins)  the  law 
enforcement  and  bomb  squad  would  initially  operate  and  ED  would  be  in  a  support  role. 
Events  are  similar  in  the  exercise.  The  two  organizations  are  completing  their  primary 
missions.  In  the  scenario  the  bomb  detonates  causing  collapse  and  fires  (phase  three).  The 
PD  exits  to  regroup  and  the  ED  enter  to  complete  fire  suppression,  assessment,  rescue  and 
collapse  shoring.  EMS  begins  play  for  victim  assessment,  triage  and  transport.  As  phase 
three  missions  are  completed,  phase  four  begins  with  investigation  and  overhaul.  As  in 
reality,  there  is  a  logical  movement  through  specific  missions  for  each  discipline.  Any 
number  of  activities  can  be  added  for  the  specific  organization.  Overlap  of  phases  can  be 


The  Phased  Response  Strategy  is  the  brain  child  of  a  Battalion  Chief  Bill  Hines  of  the  FDNY  Battalion  7,  Division  1, 
with  his  approach  of  making  exercises  beneficial  in  Penn  Station  when  operating  in  a  multi-agency  environment. 
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implemented  within  the  exercise  but  the  evaluation,  performance  measurements  and 
objectives  of  each  agency  must  be  obtained  separately.  Exercises  today  tend  to  press  all 
activities  into  one  short  time  frame.  They  create  evaluation  problems  along  with  cultural 
tensions. 

The  Phased  Response  Strategy  elongates  the  exercise  process  so  that  there  can  be 
an  exchange  of  knowledge  within  units  and  between  agencies.  Each  move  can  be 
analyzed  as  it  happens  and  recorded.  It  is  experience  that  is  learned  by  doing  and 
repeating.  The  present  exercise  system  speeds  through  the  operations  phase  so  that  they 
can  achieve  the  outcomes  before  all  the  inputs  and  outputs  are  accomplished.  That  is  the 
fallacy  of  the  present  system. 

F.  SHORTFALLS 

Despite  these  important  elements  of  progress,  the  EDNY  has  some  improvements 
that  still  need  to  be  made  to  its  exercise  program.  One  problem  is  that  many  exercises  try 
to  accomplish  too  many  objectives  simultaneously.  Learning  and  institutional  capacity¬ 
building  is  a  grassroots  phenomena  that  starts  at  the  individual  level  and  works  its  way 
horizontally  and  vertically.  Individuals  learn  and  absorb  knowledge.  Individuals  of  an 
organization  learn  and  absorb  knowledge.  Organizations  can  only  learn  and  absorb 
knowledge  because  of  the  individuals  that  are  within  the  organization.  The  main  reason 
that  major  exercises  fail  is  that  they  try  to  measure  and  evaluate  everything  that  is 
happening  within  that  exercise  in  a  short  period  of  time.  The  metrics  employed  fall  short 
of  diagnosing  an  institutional  illness  or  identifying  a  correctable  problem.  The  players  on 
a  local  level  become  disenchanted  when  large  numbers  of  other  agencies  show  up  and 
monopolize  the  activities.  This  throws  off  the  metrics  of  the  individual  and  team. 

Exercises  and  exercise  evaluation  needs  to  have  a  tiered  approach  throughout 
training.  Smaller  exercises  evaluate  the  small  unit  responses  and  their  competency. 
Larger  exercises  evaluate  the  command  and  control  aspects  of  the  operation.  If  the  units 
have  proven  their  competency  in  smaller  exercises,  then  the  factors  that  influence  the 
larger  exercises  and  push  performance  will  be  the  command  and  control  and  that  is  the 
competencies  that  should  be  measured.  Likewise,  if  an  agency  has  proven  its  ability  to 
properly  used  incident  management  skills  and  successfully  operate  at  an  incident,  then  at 
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a  multi-jurisdictional  incident,  the  emphasis  should  be  on  the  ability  of  ageneies  to 
operate  as  a  single  foree  and  the  effeets  this  eommand  has  upon  the  operational  forees. 

An  additional  problem  lies  in  the  end  of  eourse  reaetion  questionnaires  or 
“reaetionaires”  (as  they  have  eome  to  be  called)  whieh  are  used  in  many  exercises.  They 
are  intended  to  be  used  as  a  means  to  evaluate  instruetors,  eourse  eontent  and  eourse 
environment,  but  are  often  of  little  value  for  measuring  performanee.  They  do  not  inelude 
the  kinds  of  speeifie,  objeetive  metries  that  ean  be  used  to  effeetively  measure 
performanee.  At  present,  however,  the  neeessary  system  does  not  exist  to  develop  and 
evaluate  such  exercise  metrics.  The  fire  serviee  also  needs  to  establish  a  standardized 
reporting  system  for  exereises  that  quickly  provides  reports  to  both  the  Chief  Offieers  and 
members,  so  that  they  ean  evaluate  unit  performance  and  monitor  the  proeess  by  whieh 
shortfalls  in  performanee  are  remedied.  At  present,  there  is  a  general  laek  of 
aeeountability  for  both  inter-departmental  and  intra-departmental  exereises.  Who  is 
aceountable  to  train  or  retrain  members?  Aeeountability  needs  to  be  more  elearly 
defined. 

Exereises  are  an  apparatus  to  identify  the  gaps  in  training,  but  no  reliable 
meehanism  exists  to  change  training  when  performanee  gaps  are  identified.  The  Bureau 
of  Training  needs  to  be  involved  outside  of  the  training  aeademy  in  building  such  a 
mechanism.  There  is  a  definite  disconnect  between  training  and  the  operational  forees  in 
the  field.  The  Bureau  of  Training  does  a  great  job  in  making  much  of  their  training  as 
realistie  as  possible,  even  using  real  smoke  and  fire  for  some  of  their  evolutions.  This 
does  not  get  transported  over  to  exereises  whieh  still  use  artifieial  eonditions,  and  whieh 
allow  other  diseiplines  and  organizations  into  areas  that  in  reality  they  would  not  be  able 
to  venture  beeause  of  the  laek  of  personal  proteetive  gear. 

Finally,  exereises  are  always  planned,  orehestrated  events  in  whieh  partieipants 
rehearse  their  roles  prior  to  the  event.  With  the  rehearsals,  the  unrealistie  props,  the 
shortened  response  times  and  the  eompetition  for  dominanee  between  ageneies,  exereises 
eurrently  provide  a  poor  basis  to  evaluate  eapabilities  and  performanee  levels. 
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Ill:  REVIEW  OF  BEST  PRACTICES  IN  THE  U.S.  MILITARY,  THE  PRIVATE 
SECTOR,  AND  AS  DESCRIBED  IN  ACADEMIC  LITERATURE 

A,  INTRODUCTION 

The  Fire  Serviee  is  a  unique  organization  that  has  unique  requirements  and 
charaeteristics.  In  order  to  fill  the  shortfalls  identified  in  Chapter  2,  there  are  important 
lessons  that  can  be  learned  and  adopted  from  other  organizations  that  have  similar 
missions  and  have  addressed  similar  shortfalls.  The  opportunity  exists  to  take  advantage 
of  the  lessons  learned  from  the  military  and  industry  in  their  efforts  to  improve 
performance  and  couple  those  practices  with  the  best  practices  developed  within  the  fire 
service  itself. 

B.  THE  UNITED  STATES  ARMY’S  EVALUATION  AND  ASSESSMENT 

MODEL 

1.  Overview  of  the  Model 

The  United  States  Army  is  much  like  the  Fire  Service.  They  continuously  train  to 
be  prepared  for  the  battle.  When  called,  the  army  battles  an  enemy  in  a  hazardous, 
wartime  environment  and  the  Fire  Service  battles  the  ‘red  devil’,  fire,  and  a  host  of  other 
hazardous  materials  also  in  a  hazardous  environment.  When  it  comes  to  terrorism  and  the 
threat  of  WMD,  both  organizations  are  trained  to  be  proficient  in  procedures  and 
instrumentation  that,  hopefully,  they  will  never  have  to  use.  This  is  the  hardest  part  of 
preparedness;  maintaining  competency  and  proficiency  at  tasks  that  are  not  everyday 
skills.  The  only  way  they  will  know  what  level  they  are  at  or  where  they  need  to  go  is  if 
they  measure  their  performance  and  analyze  the  results.  Therefore,  the  U.S.  Army  and 
their  training  model  will  serve  as  one  of  the  foundations  for  creating  metrics  that  will  be 
useful  for  the  fire  service. 

The  Army  trains  as  they  fight,  as  realistic  as  they  can  possibly  attain.  They  assess 
every  aspect  of  their  organization  so  that  their  commander  can  judge  their  ability  to 
accomplish  their  wartime  operational  mission.  The  Army  evaluates,  assesses,  reviews. 


This  becomes  the  first  best  practice  element  for  the  Fire  Service,  the  commander  is  responsible  to  assess 
the  capability  of  his  command  through  evaluations  from  trainers,  company  commanders,  and  his  own 
knowledge  and  experience. 
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so  that  they  can  plan  and  execute  their  mission.  There  is  feedback  gathered  throughout 
the  process  that  makes  it  to  the  top  supervisors  so  that  institutional  change  can  be 
accomplished  if  needed. 


1 8 

Figure  2.1:  U.S.  Army  Training  Management  Cycle 


2,  Assessment  and  Evaluation 

The  Army  treats  everything  they  do  as  a  training  experience  and  strives  to 
evaluate  and  assess  with  the  final  objective  to  be  able  to  accomplish  their  mission  and  the 
goal  of  preparedness  24  hours  a  day,  7  days  a  week,  365  days  a  year.  The  commander  has 
the  responsibility  for  the  assessment  of  his  troops  by  using  his/her  experience,  knowledge 
of  his  troops,  personal  observation,  the  feedback  from  the  training  evaluation  reports 
from  the  units  under  his  command,  and  other  reports  and  evaluations  that  are  feed  back 
for  his  assessment.  This  assessment  is  a  continuous  process  that  includes  evaluating 
training,  conducting  an  organizational  assessment,  and  preparing  a  training  assessment 


The  next  best  praetice  is  the  feedbaek  that  is  capture  in  all  directions,  horizontally  and  vertically  in  both 
directions.  This  enables  the  program  to  adapt  to  the  environment. 

**  Training  the  Force,  p  6-1 
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which  has  become  institutionalized  within  the  foree^^.  The  measures  of  performanee  are 
evaluated  against  the  established  Army  standard  to  identify  gaps  in  performanee  and/or 
training. 

It  is  important  at  this  point  to  distinguish  between  evaluations  and  assessments. 
Evaluations  measure  past  events  in  training  so  that  a  eommander  ean  make  a  judgment  on 
his  eommands  eapability  in  the  future.  Assessments  are  a  summary  of  all  the  metries  used 
in  the  evaluations,  along  with  other  measures  to  prediet  the  future.  In  this  respeet,  the 
Army  is  using  performanee  measures  of  the  past  and  present  to  prediet  the  future. 

Everyone  and  everything  in  the  foree  is  evaluated  and  assessed  for  operational 
readiness,  from  the  lowly  private,  to  the  powerful  general  and  from  the  operations  of  the 
squad  to  the  logisties  of  their  eommand  and  eontrol.  Evaluations  used  by  the  Army  ean 
be^°: 

•  Informal  -  a  squad  leader  training  a  squad  in  taeties  with  immediate,  informal 
feedbaek.  The  metries  here  are  simple  and  usually  verbal. 

•  Eormal  -  Training  and  exereises  are  evaluated  by  dedieated  evaluators  and  are 
usually  done  by  two  ranks  above. 

•  Internal  -  are  planned,  resoureed  and  eondueted  internal  to  the  group 

•  External  -  are  planned,  resoureed  and  eondueted  by  an  eehelon  outside  and 
above  the  ehain  of  eommand 

By  using  assessments  and  evaluations  the  way  the  Army  does  in  its  training 
exereises,  they  mirror  eommon  eivilian  models  that  will  be  diseussed  in  later  seetions.  By 
evaluating  an  individual,  group,  team  or  unit,  the  eommander  evaluates  this  speeifie 
training  audienee’s  performanee  as  it  relates  to  the  eompletion  of  their  speeifie  task. 
When  the  eommander  uses  the  assessment  proeess,  he  embraees  the  eompilation  of 
training  evaluation  reports  from  individual  units  and  groups,  feedbaek  from  the  multiple 
levels  of  evaluators  and  experienee  from  personal  observation  and  knowledge  of  the 
training,  so  that  he  ean  prediet  if  his  eommand  is  able  to  work  jointly  to  aehieve  it’s 
objeetives.  In  other  words,  the  eommander  of  the  troops  ultimately  assesses  whether  or 
not  his  eommand  is  prepared  and  what  level  of  preparedness  the  eommand  has  obtained. 

Training  the  Force,  Chapter  6,  p  6-1 

Training  the  Force,  (FM  7-0,  Field  Manual,  2002),Chapter  6,  p6-3 
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3,  After  Action  Reviews  (AAR) 

2 1 

The  Army  conducts  an  After  Action  Review  (AAR)  after  every  training  event. 
Feedback  is  their  main  assessment  tool.  The  AAR  is  a  method  of  providing  feedback  to 
units  by  involving  participants  in  the  training  diagnostic  process  in  order  to  increase  and 
reinforce  learning.  The  AAR  leader  guides  participants  in  identifying  deficiencies  and 
seeking  solutions  .  The  AAR  can  be  formal  or  informal,  structured  or  unstructured,  and, 
most  important  of  all,  it  is  not  a  critique.  The  AAR  is  the  Army’s  most  used  and 
important  metric  for  learning  and  it  has  the  following  advantages  over  a  critique  : 

•  It  focuses  directly  on  key  METL^"^  derived  training  objectives 

•  Emphasizes  meeting  Army  standards  rather  than  pronouncing  judgment  of 
success  or  failure 

•  Uses  “leading  questions”  to  encourage  participants  to  self-discover  important 
lessons  from  the  training  event 

•  Allows  a  large  number  of  individuals  and  leaders  to  participate  so  more  of  the 
training  can  be  recalled  and  more  lessons  learned  can  be  shared. 


The  AAR  consists  of  four  parts: 

1 .  Review  what  was  supposed  to  happen  (training  Plan). 

2.  Establish  what  happened 

3.  Determine  what  was  right  or  wrong  with  what  happened. 

4.  Determine  how  the  task  should  be  done  differently  the  next  time. 


The  key  to  fighting  and  winning  is  an  understanding  of  ‘how  we  train  to 
fight’  at  every  echelon.  Training  programs  must  result  in  demonstrated 
tactical  and  technical  competence,  confidence,  and  initiative  in  our 
soldiers  and  their  leaders.  Training  will  remain  the  Army’s  top  priority 
because  it  is  the  cornerstone  of  combat  readiness! 

25 

-General  Carl  E.  Vuono 


Training  the  Force,  Chapter  6,  p  6-4 

Definition  from  Training  the  Force,  Glossary,  p  8 

Training  the  Force,  (FM  7-0,  Field  Manual,  2002),Chapter  6,  p  6-4 

METL  (mission  essential  task  list)  is  a  compilation  of  collective  mission  essential  tasks  an  organization 
must  perform  successfully  to  accomplish  its  wartime  mission(s)  [Training  the  Force,  Glossary,  pl3]. 
METL’s  are  very  similar  to  competencies  used  in  the  fire  service. 
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4,  The  Army’s  Ten  Principles  of  Training 

The  Army  model,  Training  the  Foree,  (FM  7-0,  Field  Manual,  2002),  serves  their 
organization  and  their  objectives  well.  The  Army  has  Ten  Principles  of  Training^^: 

•  Commanders  are  responsible  for  training 

•  NCO’s  train  individuals,  crews  and  small  teams 

•  Train  as  a  combined  arms  and  joint  team 

•  Train  for  combat  proficiency 

•  Realistic  conditions 

•  Performance-oriented 

•  Train  to  Standard  Using  Appropriate  Doctrine 

•  Train  to  Adapt 

•  Train  to  Maintain  and  Sustain 

•  Train  Using  Multi-echelon  Techniques 

•  Train  to  Sustain  Proficiency 

•  Train  and  Develop  leaders 

5,  The  U,S.  Army’s  Train  the  Force  Model  -  Good  Points,  Bad  Points 
and  Its  Implications  for  the  Fire  Service 

The  Army’s  model  serves  as  a  template  for  the  entire  military.  Though  they  lack 
actual  quantifiable  metrics,  they  do  a  good  job  of  limiting  outside  influences  on  their 
qualitative  metrics.  This  allows  the  final  assessment  of  readiness  (preparedness  for  the 
Fire  Service)  to  be  a  very  objective  decision  by  the  commander  and  it  can  leave  room  in 
the  final  formula  to  make  very  general  assumptions.  If  the  commander  is  an  authoritative 
personality  that  prefers  to  make  decisions  on  his  own  without  consultation  with  qualified 
peers,  the  results  could  be  deadly. 

The  military  culture  of  a  supreme  hierarchy  and  obedience  to  authority  limits 
potentially  corruptive  cultural  influences  that  the  civilian  world  must  endure.  Certain 
biases  and  poor  work  ethics  have  little  place  in  the  military  thinking  and  thus  do  not  exert 
the  pressures  as  they  do  in  the  civilian  world. 


This  quote  can  readily  be  adapted  to  the  new  mission  that  must  be  performed  by  the  Fire  Service. 
Training  the  Force,  (FM  7-0,  Field  Manual,  2002),  pp.  2-2  through  2-9. 
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The  concepts  that  should  be  mirrored  in  the  fire  service  when  developing  metrics 
for  performance  are; 

•  Evaluation  and  Assessment  -  the  use  of  evaluations  to  measure  the  past  and 
present  in  order  to  assess  the  future  capabilities.  The  Fire  Service  uses 
evaluations  to  improve  training  and  improve  the  performance  of  the  members. 
A  disconnect  occurs  when  these  evaluations  are  not  forwarded  to  the 
leadership  for  assessment  of  the  training  and  eventual  assessment  of 
preparedness. 

•  Hierarchical  Involvement  -  the  involvement  of  the  highest  levels  of  the 
organization  in  the  training  process.  The  Fire  Service  hierarchy  generally  is  to 
far  distant  from  the  actual  process.  The  military  practice  to  train  one  level 
down  and  evaluate  two  levels  up  will  bring  the  hierarchy  a  step  closer  to  the 
operational  building  block  -  the  unit. 

•  Accountability  -  their  ability  to  hold  personnel  and  their  leaders  at  every  level 
accountable  for  training.  The  commander  is  responsible  for  the  training  and 
performance  of  members  but  the  NCO  does  the  actual  training  and  the  NCO  is 
held  strictly  accountable  by  the  commander.  The  Fire  Service,  to  the  contrast, 
also  makes  the  company  commander  of  the  unit  the  responsible  party  for 
training  but  is  consistently  very  lax  in  holding  that  same  commander 
accountable. 

•  Assessment  of  Readiness  by  tbe  Commander  (only)  -  The  commander  is 
solely  responsible  for  the  assessment  of  the  troops  under  him  in  the  military 
model.  This  practice  works  well  in  the  military  because  they  can  control 
outside  influences  on  the  decision  process  but  is  a  dangerous  practice  for  the 
Fire  Service.  This  thesis  would  support  an  Executive  Committee  for 
Preparedness  Evaluation  (ECPE)  who  would  have  the  task  of  assessing  the 
preparedness  levels  of  units. 


Training  the  Force,  (FM  7-0,  Field  Manual,  2002),  p  2-2 
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C.  BEST  PRACTICES  FROM  THE  PRIVATE  SECTOR 

1.  Justification  for  Examining  Industrial  Models 

The  Fire  Serviee  uses  metrics  to  measure  the  performance  of  their  members  to 
establish  their  level  of  preparedness  to  react  to  a  situation  when  called  upon  to  respond. 
The  military  uses  metrics  to  measure  the  performance  and  capability  of  their  members  so 
that  a  commander  can  predict  the  readiness  of  the  command  to  complete  a  mission.  So 
too,  the  private  sector  uses  metrics  to  measure  every  aspect  of  their  performance,  from 
the  resource  procurement  and  manufacturing  process  to  the  sales  and  distribution  of  a 
product  to  increase  the  profit  (the  bottom  line).  The  motive  does  not  matter,  the 
performance  measurement  principles  are  the  same  for  all  three  organizations.  The 

literature  deals  more  with  industry  then  any  other  discipline.  The  common  concept 
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throughout  the  literature  is,  “you  can’t  improve  what  you. .  .don’t  measure”. 

2.  Performance  Measurements  Models 

This  section  will  look  at  select  models  for  performance  measurement  that  have 
been  developed  through  the  years  and  vary  in  small  respects  according  to  the  emphasis  of 
the  particular  authors.  Table  1  ,  taken  from  a  report  written  by  Tamkin,  Yarnell  and 
Kerrin,  for  the  Institute  for  Employment  Studies  (IBS)  of  the  United  Kingdom,  shows  a 
comparison  of  eight  models  that  are  in  use  across  the  private  sector  to  evaluate  training 
and  performance  within  specific  organizations.  The  models  are  similar  but  vary  according 
to  the  specific  application  they  are  used  to  evaluate.  Some  of  the  models  diverge  from  the 
original  in  their  approach  to  how  training  evaluation  might  take  place. 


a.  Kirkpatrick  -  Four  Step  Model 

Training  evaluation  is  a  bit  like  eating  five  portions  of  fruit  and  vegetables 
a  day;  everyone  knows  that  they  are  supposed  to  do  it,  everyone  says  they 
are  planning  to  do  better  in  the  future  and  few  people  admit  to  having  got 
it  right. 

-Tamkin,  Yarnell,  Kerrin 


Harbour,  Jerry  L.,  PhD,  The  Basics  of  Performance  measurement,  Productivity  Press,  1997,  p.  1 
Tamkin,  P.,  Yarnell,  J.,  Kerrin,  M.,  Kirkpatrick  and  Beyond:  A  review  of  models  of  training  performance, 
IBS,  2002,  p  13 
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The  first  and  original  model  is  by  Kirkpatriek  who  is  considered  to  be  the  father 
of  modern  evaluation  measurement  models  because  his  model  forms  the  basis  for  all  that 
have  followed.  In  1959  he  described  four  steps,  or  levels,  as  they  later  became  known, 
that  you  can  evaluate  in  order  to  measure  whether  a  training  program  or  an  exercise  is 
affective  and  adhering  to  the  strategic  plan  of  the  organization  : 

•  Level  1:  Reaction  -  what  the  participants  thought  of  the  training,  normally 
measured  in  a  questionnaire  format. 

•  Level  2:  Learning  -  the  changes  in  knowledge,  skills,  or  attitude  with  respect 
to  the  training  objectives,  normally  assessed  by  use  of  performance  tests. 

•  Level  3:  Behavior  -  changes  in  job  behavior  resulting  from  the  program,  to 
identify  whether  the  learning  is  being  applied.  Assessment  methods  include 
observation  and  productivity  data. 

•  Level  4:  Results  -  the  bottom-line  contribution  of  the  training  program. 
Methods  include  measuring  costs,  quality  and  return  on  investment  (ROI). 

Kirkpatrick’s  model  is  simple,  easy  and  sensible  as  compared  to  some  of  the 
newer  models  and  it  is  readily  adaptable  to  the  fire  service  to  produce  performance 
measures  for  exercise  evaluation.  He  has  been  criticized  for  a  too  simplistic  view  that 
fails  to  take  into  account  the  multiple  variables  that  influence  the  transfer  of  knowledge 
from  the  starting  point  on  level  1  to  the  results  that  actually  produce  institutional  change 
within  the  organization  because  of  the  training.  The  metrics  that  can  be  applied  to  this 
model  are  simple  and  easy  to  use.  The  questionnaire  and  examination  measures  for  level 
1  and  2  are  in  common  use  today  within  the  fire  service  and  provide  both  qualitative  and 
quantitative  data  points  to  make  decisive  decisions  in  quality,  specificity,  and 
effectiveness  of  training.  Metrics  can  easily  be  applied  to  gain  important  data  from  level 
3  and  level  4  but  have  inherent  pitfalls  depending  on  the  measures  used  for  this  purpose. 


Kirkpatrick  developed  his  four  step  model  in  1959  and  provided  a  simple  and  pragmatic  model  for 
helping  practioners  think  about  training  programs.  The  majority  of  the  work  that  followed  is  based  upon 
Kirkpatrick’s  program,  (excerpt  from  Tamkin,  P.,  Yamell,  J.,  Kerrin,  M.,  Kirkpatrick  and  Beyond:  A 
review  of  models  of  training  performance,  IBS,  2002,  p.  ix) 

Kirkpatrick  and  Beyond,  p  3 
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b.  Hamblin  -  Five  Step  Model 

The  first  modification  of  Kirkpatrick’s  model  was  described  by  Hamblin 
(1974)  in  a  five  level  model.  He  uses  Kirkpatrick’s  original  first  three  levels  and  expands 
upon  the  fourth  level  to  account  for  changes  training  can  have  on  the  organization  and 
upon  the  financial  bottom  line  of  the  organization  and  the  economy  as  a  whole.  He 
suggests  that  his  model  forms  a  hierarchy  of  training  evaluation  and  that  the  levels  can 
form  an  ascending  value  of  information  with  the  final  level  being  ultimately  the  most 
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important  outcome,  financial  effects. 

Both  authors  have  been  criticized  (Holton  1996)  for  not  accounting  for 
intervening  variables  that  effect  learning  and  the  knowledge  transfer  process.  Holton 
further  states  that  no  model  can  be  validated  without  measuring  outcomes  and  developing 
metrics  for  that  purpose.  Kraiger  and  Jung  (in  Quinones,  1997)  argues  that  Kirkpatrick’s 
model  serves  as  a  framework  of  how  to  evaluate  training  but  does  little  on  what  to 
evaluate  and  how  to  link  the  results  to  the  strategy  of  the  organization.  All  the  criticism  of 
Kirkpatrick’s  model  seems  to  be  a  constructive  exercise  that  allows  the  model  to  be  the 
foundation  from  which  many  other  models  have  been  built. 

c.  Organizational  Elements  Model  ( OEM)  -  Six  steps 

The  Organizational  Elements  Model  (OEM)  suggested  by  Kaufman  and 
Keller  (1994)  expands  on  Kirkpatrick’s  model  to  discuss  societal  influences  upon 
outcomes.  They  argue  that  each  level  should  include  planning  and  needs  assessment  and 
an  examination  of  desired  or  expected  results.  This  would  improve  the  quality  of  data 
measured  so  a  more  accurate  analysis  could  result  in  a  more  useable  outcome. 

In  1995,  with  the  help  of  Watkins  ,  Kaufman  and  Keller  described  six 

levels:^"^ 

•  Level  1:  Input  -  similar  to  Kirkpatrick’s  reaction  level  but  has  been  expanded 
to  include  the  role,  usefulness,  appropriateness  and  contribution  of  the 
methods  and  resources  used 


Tamkin,  P.,  Yamell,  J.,  Kerrin,  M.,  Kirkpatrick  and  Beyond:  A  review  of  models  of  training  performance, 
IBS,  2002,  p5 

Watkins  helped  the  team  of  Kaufman  and  Keller  produce  the  six  step  model, 
ibid,  Tamkin,  p  6 
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•  Level  2:  Process  -  the  reaction  level,  but  also  includes  an  analysis  of  whether 
the  intervention  was  implemented  properly  in  terms  of  achieving  its  objective 

•  Level  3:  Micro  (acquisition)  -  the  learning  level  plus  it  examines  the 
individual  as  well  as  small-group  mastery  and  competence 

•  Level  4:  Micro  (performance)  -  links  closely  to  behavioral  level  and 
examines  the  utilization  of  skills  and  knowledge.  This  focus  is  on  application 
rather  than  transfer  of  skills  and  knowledge 

•  Level  5:  Macro  -  relates  to  the  results  level  and  examines  organizational 
contributions  and  payoffs 

•  Level  6:  Mega  -  an  additional  level  which  looks  at  societal  outcomes 


d.  Indiana  University  and  the  Carousel  of  Development: 

Industrial  Society  Model  (IS) 

Two  other  models,  the  Indiana  University  approach  (Molenda,  Pershing 
and  Reigheluth,  1996)  and  the  Carousel  of  Development  (Industrial  Society  2000) 
continue  to  refine  the  four  levels  with  differing  emphasis  upon  specific  levels.  Molenda, 

-5C 

et  al,  describes  six  ‘Stratum’  and  puts  emphasis  on  the  ‘impact  that  changed 
performance  will  have  on  society’  while  the  Industrial  Society,  emphasizing  six  stages, 
aims  to  validate  and  evaluate  training.  “Andrew  Forest  of  the  Industrial  Society  argues 
that  ‘true  evaluation  needs  to  take  place  long  before  and  after  training  has  taken  place’.” 


e.  Return  on  Investment  (ROI)  and  KPMT^^  Models 
The  five-level  Return  On  Investment  (ROI)  model  (Phillips,  1994,  Phillips 
and  Holton,  1995)  attempts  to  use  the  Kirkpatrick  four-level  model  and  add  a  link  to 
monetary  benefits  measuring  the  relationship  of  training  results  to  cost  or  what  is  the 
return  on  the  investment  we  put  into  the  training?  This  line  of  thought  is  expanded  with 
the  KPMT  model  (Kearns,  P.  and  Miller,  T.,  1997).  They  argue  that  pre-measurements 
need  to  be  in  place  to  assess  if  a  training  program  is  appropriate.  These  measurements 


ibid,  Tamkin,  p  7 
ibid,  Tamkin,  p  7 
ibid,  Tamkin,  p  8 

ibid,  Tamkin,  p  8,  KPMT  is  the  first  letter  of  the  last  and  first  names  of  the  authors,  Kearns,  P.  &  Miller, 
T. 
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identify  the  need  and  the  training  program  is  developed  for  that  need.  The  program  can 
only  bring  added  value  to  the  organization  if  there  is  poor  performance  or  there  is  a  need 
identified  to  further  the  mission  statement  of  the  organization.  In  other  words,  an 
organization  should  not  train  for  the  sake  of  training,  there  must  be  an  identifiable  need 
and  the  use  of  metrics  can  be  instrumental  in  the  identification  process. 

f  CIRO  -  Context,  Input,  Reaction,  Outcome  Model 

The  CIRO  (Context,  Input,  Reaction,  Outcome  Approach)  model  (Warr, 
Bird,  and  Rackham,  1970)  restructures  the  four  level  approach  and  front  loads  the  process 
by  analyzing  the  context  and  inputs  to  the  training  program  prior  to  assessing  the  reaction 
level. 


g.  Alternative  Evaluation  Models 

This  section  will  embrace  alternate  model  areas,  models  that  focus  on 
evaluation  and  models  using  different  types  of  measures.  These  models  stray  away  from 
the  four-level  Kirkpatrick  model  and  emphasize  other  critical  areas. 

Pulley  (1994)  described  a  model  that  places  evaluation  and  metric 
development  as  being  dependent  upon  what  is  good  for  the  end  user.  He  uses  stages  to 
accomplish  his  argument  : 

1.  Identify  the  decision-makers  so  as  to  ascertain  who  will  be  using  the 
information  and  what  their  stake  in  it  is. 

2.  Identify  the  information  needs  of  the  decision-makers  -  what  do  they  need  to 
know  and  how  will  it  influence  their  decisions? 

3.  Systematically  collect  both  quantitative  and  qualitative  data. 

4.  Translate  the  data  into  meaningful  information. 

5.  Involve  and  inform  decision-makers  on  an  on-going  basis. 

One  of  the  alternative  models  describes  four  different  educational 
evaluation  processes  (Stuffiebeam  et  ah,  1971). 

1 .  Context  Evaluation  -  which  helps  in  planning  and  developing  objectives 

2.  Input  evaluation  -  which  helps  to  determine  the  design  by  examining 
capability,  resources  and  different  strategies. 

3.  Process  evaluation  -  which  helps  to  control  the  operations  by  providing  on¬ 
going  feedback. 


ibid,  Tamkin,  p  15 
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4.  Product  evaluation  -  which  helps  to  judge  and  reaet  to  the  program 
attainments  in  terms  of  outputs  and  outeomes. 

Two  other  alternative  models  eontinue  to  emphasize  evaluation  as  a  foeal 
point.  Newby  (1992)  argues  that  evaluations  ean  and  should  be  done  during  training  and 
exereise  events  and  after  the  event  in  the  work  environment,  by  using  performanee 
metries.  He  also  argues  that  the  eriteria  to  measure  ean  be  abstraet  influenees  that  affeet 
the  ability  to  perform,  sueh  as  soeietal,  moral,  politieal  or  philosophieal  eriteria."^*’  Preskill 
and  Torres  (1999)  emphasize  evaluation  is  a  learning  proeess  that  eonnects  the 
organizational  mission  and  strategie  plans.  They  emphasize  issues  eritieal  to  the 
organization,  promote  dialogue,  emphasize  internal  proeesses  as  well  as  outcomes, 
provide  educational  and  training  opportunities,  promote  eollaboration,  eo-operation  and 
participation,  and  use  diversity  of  perspeetives  to  develop  understanding  about 
organizational  issues.  They  treat  this  proeess  as  a  learning  opportunity  for  the  entire 
organization. 


h.  Models  Using  Different  Measures:  Evaluation  Models 
Kraiger  et  al.  (1993)  argues  that  evaluation  measures  will  beeome  elearer 
if  we  link  learning  outcomes  and  training  evaluation.  They  distinguish  between  three 
different  types  of  training  outeomes,  eognitive,  skill-based,  and  affeetive.  If  we  take  these 
and  look  at  them  from  the  perspeetives  sueh  as  the  goals  of  the  training,  the  proeess 
strategies  and  the  performance  criteria,  what  the  evaluation  measures  will  be  and  their 
results  will  be  clearer  and  more  useful  to  the  end  user. 

The  one  alternative  model  that  has  gained  wide  aeeeptanee  throughout  the  private, 
eorporate  world  in  the  last  eight  years  is  the  Balaneed  Seoreeard  model  (Kaplan  and 
Norton,  1996).  They  use  measures  aeross  four  areas  of  an  organization:  finanee, 
eustomers,  internal  business  proeesses,  and  learning  and  growth,  with  the  first  set  of 
measures  for  finanee  being  as  important  as  the  metries  for  learning  and  growth. 

3,  Summary:  Private  Sector  Concepts  That  Can  Be  Adapted  to  the  Fire 
Service 


ibid,  Tamkin,  p  16 
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a.  Kirkpatrick’s  model  describes  four  levels  of  evaluation  that  should  be 
undertaken.  The  fire  service  measures  two  of  them  routinely,  the  reaction  level  and  the 
learning  level.  A  large  portion  of  our  training  regiment  is  classroom,  instructor  based  and 
evaluation  questionnaires  evaluating  the  course  material  and  level  of  instruction  are  easy 
and  commonplace  along  with  post-testing.  Neither  behavioral  change  nor  results  of  the 
training  are  officially  evaluated  on  a  professional  level.  But,  through  the  use  of  exercises, 
field  evaluation  of  skill  levels  of  individuals,  units  and  multi-unit  interaction  can  be 
observed.  Also,  Newby  (1992)  points  out,  that  evaluations  should  continue  after  the 
training  to  ensure  the  effectiveness  of  the  training.  This  is  one  concept  that  needs  to  be 
adapted  widely  by  the  fire  service  if  our  level  of  preparedness  to  respond  to  terrorist 
incidents  that  are  few  and  far  between  is  to  be  optimized. 

b.  The  private  sector  reinforces  the  military  concept  of  evaluation  and  assessment 
to  bring  the  fire  service  to  an  anticipated  level  of  preparedness.  Both  the  IS  Carousel  and 
the  KPMT  models  point  out  the  necessity  to  identify  the  need  through  measures  that  point 
out  deficiencies  and  gaps.  This  will  propel  us  into  the  learning  process  where  constant 
evaluation  and  measures  need  to  be  used  to  benchmark  our  progress.  We  must  also  take 
into  account  the  individual  factors  that  will  influence  our  learning,  positively  or 
negatively,  and  exploit  them.  This  will  result  in  outcomes  that  need  to  be  reinforced  by 
the  organization  to  create  behavioral  change.  Each  of  these  levels  can  be  measured  and 
benchmarked  to  keep  the  process  moving  forward  and  to  provide  feedback  to  improve 
process.  With  this  accomplished,  real,  positive  organizational  change  can  be  measured 
and  should  be  measured. 

c.  Pulley  (1994)  suggests  that  we  should  identify  the  stakeholders  and  ascertain 
their  needs.  We  should  not  use  metrics  for  the  sake  of  measuring:  they  should  be 
purposeful,  exact  and  sensible,  for  the  end-user.  Measures  should  also  be  simple  and 
translate  easily  into  meaningful  data  and  should  be  both  qualitative  and  quantitative  in 
nature.  The  metrics  should  paint  a  picture  of  the  training  and  the  preparedness  of  the  unit 
in  the  eyes  of  the  commander  so  that  he  could  use  it  to  assess  their  true  preparedness 
capability.  . 
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IV:  DEVELOPMENT  OF  METRICS 


A.  INTRODUCTION 

Before  determining  which  performance  metrics  that  should  be  incorporated  into 
the  FDNY  exercise  system,  we  need  to  explore  the  criteria  by  which  such  decisions 
should  be  made.  The  following  chapter  examines  the  specific  objectives  that  need  to  be 
accomplished  by  metrics,  and  applies  them  to  key  issues  in  the  FDNY  exercise  system, 
with  reference  to  the  shortfalls  and  best  practices  that  were  identified  by  the  preceding 
two  chapters. 

B,  CAPABILITY,  PREPAREDNESS  AND  READINESS 

Capability,  Preparedness  and  Readiness  are  three  terms  that  are  often  used 
together  but  are  not  interchangeable.  These  are  three  very  important  concepts  for  the  Fire 
Service  when  it  comes  to  the  application  and  analysis  of  the  metrics  used  during  training 
and  exercises. 

Capability  refers  to  the  ability  of  a  member  or  unit  to  effectively  complete  a  task. 
The  Fire  Service  has  many  units  with  differing  capabilities.  An  engine  company  has  the 
capability  to  stretch  hose  lines  and  pump  water  thus  extinguishing  the  flames.  A  truck 
company  has  the  ability  to  vent,  enter  and  search  (VES)  thus  venting  the  fire  building, 
causing  conditions  to  improve  to  increase  survivability  of  rescuers  and  victims,  forcing 
entry  to  a  building  to  allow  the  stretching  of  the  line  and  searching  for  the  origin  of  the 
fire  and  rescue  of  victims.  The  units,  operating  together  as  a  team,  have  the  capability  to 
extinguish  the  fire.  Thus,  it  is  important  for  the  Incident  Commander  to  know  the 
capability  of  the  responding  companies  to  assign  tasks. 

Preparedness  is  the  state  of  full  readiness."^'  The  Fire  Service  is  being  prepared  to 
increase  its  capability  to  operate  at  terroristic  events  through  training,  equipment 
familiarization  and  exercise  evaluation.  It  is  this  preparedness  that  the  Fire  Service  needs 
to  analyze  through  the  use  of  metrics  in  order  to  have  an  idea  of  tasks  and  missions  it  will 
be  able  to  support,  the  budgets  it  will  need  and  the  manpower  it  will  take  to  complete  its 
sphere  of  operations. 

Encarta  online  dictionary,  www.encarta.msn.com/encnet/features/dictionarv.  May  2,  2004 
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Readiness  is  the  state  of  being  prepared  for  something. In  order  to  be  ready  to 
attempt  a  mission,  you  need  to  be  prepared  to  handle  the  outeome  of  your  actions.  If  you 
are  at  heightened  readiness,  then  you  are  prepared  to  handle  many  outcomes.  Readiness 
presupposes  some  kind  of  preparedness.  All  three  of  these  concepts  are  measurable 
through  performance.  The  measurements  can  be  quantified  or  can  simply  be  observed  by 
qualified  experts. 

C.  MEASURING  PERFORMANCE 

Performance  measure  is  the  single  most  important  metric  to  gather  in  that  it 
measures  inherent  capability.  Capability  is  at  the  heart  of  preparedness.  A  commander 
must  know  how  his  command  performs  in  order  to  assess  their  capability.  This  capability 
can  determine  their  preparedness  and  readiness  so  that  their  future  performance  can  be 
predicted.  In  this  manner,  the  commander  is  using  metrics  to  measure  the  past 
performance  so  that  he  can  measure  capability  to  assess  preparedness  and  readiness  at  the 
present  time  so  that  he  can  predict  future  performance. 

,  .you  have  to  know  where  your  going  and  where  you ’ve  been  to  plan 

how  to  get  there. . .  ”  -Brown,  et,  al,  (1996) 

Designing  the  proper  measuring  system  for  an  organization  and  selecting  the 
measures  to  use  will  involve  answering  five  different  types  of  questions. 

1,  Why  Measure? 

You  can’t  improve  what  you  can’t  (or  don’t)  measure  -  Jerry  Harbour 

The  development  and  use  of  measures  for  the  specific  objectives  and  goals  of  the 
course  or  exercise  is  tantamount  to  producing  a  worthwhile  and  successful  training 
program.  The  following  are  reasons  why  better  measures  must  be  developed  by  FDNY ; 

•  You  cannot  improve  if  you  do  not  measure. 

•  Identify  deficiencies  and  strengths  in  training  protocols. 

•  Improve  training  protocols  and  plan  for  future  training  needs. 

Cambridge  Dictionary  Online,  http://dictionarv.cambridge.org.  Mat  2,  2004 
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•  Measure  competeney,  performanee  and  proficiency  for  mission  essential 
tasks. 

•  Use  as  a  basis  to  institute  real  Institutional  Change. 

•  Integral  and  essential  part  of  the  training  cycle. 

•  Provide  superiors  with  a  measurement  on  the  success  of  their  programs. 
Legitimize  the  training. 

•  Comparison  to  other  exercises. 

•  Change  the  old  way  of  training  in  the  fire  service. 

2.  Who  Do  You  Measure? 

What  gets  measured,  gets  done.  -  Frost,  Robert 
a.  The  Individual  Participants 

The  main  focus  of  small  exercises  or  training  scenarios  is  the  individual 
participants.  These  are  usually  end  of  course  training  and  evaluation  exercises  run  on 
specific  scenarios  by  the  instructors.  As  the  exercise  grows  in  scale  the  emphasis  the 
focus  should  shift  to  the  group  or  the  team.  Here  we  can  make  a  distinction  between  a 
group,  which  is  an  association  of  individuals  whose  functions  are  interchangeable  and  are 
brought  together  for  a  specific  purpose  and  disbanded  when  the  task  is  achieved."^^  A 
team"^"^  is  different.  It  is  two  or  more  people  with  different  tasks  who  work  together 
adaptively  to  achieve  specified  and  shared  goals.  The  central  feature  is  coordination.  The 
team’s  functional  requirements  always  include  simultaneity"^^,  sequencing"^^  or  both. 
Teams  usually  have  a  history  and  a  future  and  team  performance  models  agree  that  there 
are  three  levels  of  analysis  for  teams:  individual,  team,  and  organizational."^^ 

The  question  of  who  do  you  measure  during  an  exercise  depends  upon  what  your 
objectives  are  as  it  relates  to  your  overarching  strategy.  The  fire  service  would  need  to 
measure  several  different  groups  depending  on  the  goals  and  objectives  of  the  exercise. 

Brannick,  Michael  T. 

Brannick,  Michael  T. 

Simultaneity  means  the  team  members  must  do  something  at  the  same  time  such  as  play  notes  at  the  same  time 

Sequencing  means  that  the  output  of  one  team  member’s  task  or  tasks  is  the  input  to  another  team  member’s  task. 

Brannick,  Michael  T. 
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The  individual  participant  should  be  measured  if  competency  and  proficiency  are 
important  for  the  exercise.  The  middle  level  operational  supervisor  and  the  command 
chiefs  would  be  measured  if  command  and  control  issues  need  to  be  developed  or 
improved.  The  interactions  of  agencies  or  jurisdictions  are  very  important  for  the 
outcome  of  the  exercise.  The  measurement  of  the  group  and  team  dynamic  should  be 
assessed  because  of  the  influence  this  dynamic  has  on  the  success  of  an  exercise. 

b.  Groups  and  Teams 

It  is  important  to  distinguish  between  a  team  and  a  group.  The  fire  service 
operates  in  teams  of  two  to  six  members,  all  performing  a  different  function  in  order  to 
complete  an  objective.  The  whole  team  performs  as  one.  Metrics  that  measure  how  a 
team  operates  or  how  the  individual  operates  within  the  team  would  be  valuable  to 
predict  their  performance. 

Groups  operate  in  pairings  of  two  or  more  or  groups  of  groups  that  all  perform  the 
same  function  to  complete  a  task.  Law  enforcement  and  emergency  medical  personnel 
generally  operate  as  groups  of  teams  that  perform  a  common  task  to  complete  an 
objective.  The  normal  pairing  is  two  individuals  that  form  a  team  and  then  two  or  more  of 
those  teams  form  a  group.  Metrics  that  measure  the  interaction  of  that  group  as  they  relate 
to  performing  an  overall  objective  would  be  important  to  analyze  when  placing  these 
groups  together  in  the  real  operational  world.  If  they  perform  well  during  an  exercise  that 
is  well  planned  and  executed,  then  it  can  be  assumed  that  their  performance  would 
continue  under  operational  conditions. 

c.  Supervisors  and  Incident  Commanders 

Supervisors  and  Incident  commanders  are  also  a  target  of  exercise  metrics. 
Multi-agency  operations  and  drills  are  interactions  of  differing  groups  of  responders 
operating  together  to  achieve  an  outcome.  Exercises  not  only  measure  individual 
performance  of  operational  teams,  they  also  measure  the  actions  of  supervisors.  Were  the 
supervisors  able  to  establish  effective  command  and  control  over  the  operation?  Did  the 
commanders  instill  trust  and  confidence  in  the  operational  members?  Measures  would  be 
very  valuable  to  analyze  if  they  captured  the  correct  metrics  from  below,  horizontally  or 
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from  above.  Was  incident  command  procedures  implemented  and  if  so  were  they 
effectively  activated? 

3.  What  Do  You  Measure? 

Not  everything  that  can  be  counted  counts,  and  not  everything  that  counts 

48 

can  be  counted.  -  Albert  Einstein 

The  Army  calls  the  ability  of  individual  operators  to  perform  operations  that  are 
vital  for  the  completion  of  their  objectives  or  team  objectives  as  ‘mission  essential  tasks’. 
The  fire  service  refers  to  this  ability  as  ‘competencies’.  The  ability  to  read  and  interpret 
readings  on  a  meter,  or  the  ability  to  operate  a  hand  held  assay,  would  definitely  have  an 
effect  on  the  outcome  on  a  situation.  These  would  be  measured  at  smaller  training 
exercises  or  in  training  drills  but  are  a  main  piece  of  the  puzzle  for  overall  performance. 

Don’t  measure  what  you  can’t  or  won’t  use.  -  Harbour,  Jerry  L. 

A  critical  enabler  in  achieving  preparedness  levels  and  the  desired  level  of 
performance  from  emergency  responders  is  the  ability  to  measure  performance.  In  order 
to  begin  the  measuring  process,  you  must  know  where  you  are  at  the  present  time.  You 
must  establish  a  baseline,  a  beginning,  a  starting  point.  Once  you  know  where  you  are, 
you  can  establish  where  you  want  to  go.  The  direction  you  want  to  take  the  emergency 
responders  along  their  line  to  preparedness.  Once  you  have  established  these  two  points 
you  can  determine  the  delta  or  the  gap  between  baseline  and  goal.  Develop  and 
implement  a  process  that  is  consistent  with  the  strategy  of  your  organization  that  will  get 
you  to  your  goal  and  close  the  gap.  It  is  often  helpful  to  establish  benchmarks  of 
achievement  to  map  out  the  path  or  re-measure  periodically  using  the  same  method  you 
used  to  establish  the  baseline. 

The  use  of  performance  metrics  during  this  process  can  be  simple  measures  that 
establish  the  baselines  like  the  measurement  of  time  factors,  such  as  how  long  it  takes  to 

Quote  by  Albert  Einstein  taken  from  a  NFS  thesis  by  Leighty,  James  E.,  Deeember  2001,  p.  13. 
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get  responders  onto  the  scene  of  an  incident.  Alternatively,  the  measure  can  be  numerical 
(e.g.,  how  many  responders  responded  to  the  incident?)  If  you  take  a  baseline  and  at  any 
particular  incident  you  find  that  it  takes  six  minutes  to  get  an  ambulance  to  the  scene  of  a 
particular  incident  then  you  have  identified  a  gap  in  response.  You  set  a  goal  of  three 
minutes  for  a  maximum  response  and  then  set  about  producing  a  process  to  achieve  that 
goal.  An  exercise  can  be  developed  to  test  a  realistic  response  time  and  measured.  An 
ambulance  is  set  at  point  A  and  it  needs  to  respond  to  point  B.  If  you  take  into  account  all 
the  variables  and  control  them  so  that  they  are  minimized  in  the  exercise  and  you  achieve 
the  three  minute  goal,  then  you  know  it  is  realistic  and  you  strive  to  reproduce  that  in  the 
real  operational  atmosphere.  If  you  find  that  you  can  not  match  the  goal  in  this  exercise, 
then  you  need  to  identify  the  factors  that  are  influencing  this  operation  and  change  it  by 
reducing  its  influence  or  creating  new  factors  that  have  less  of  an  influence.  That  is 
creating  real  institutional  change  on  a  simplified  level. 

What  we  are  actually  doing  is  setting  a  process  for  the  transformation  and 
blending  of  a  set  of  inputs  (emergency  responders  with  their  skill  sets,  including  control 
of  the  incident,  mitigation  of  the  hazard,  rescue  of  the  victims)  into  a  more  valuable  set  of 
outputs  (protection  of  life  and  property).  This  process  can  be  further  subdivided  into  a 
series  of  interrelated  activities  (instrument  usage  for  identification  of  CBRNE,  over¬ 
packing  of  dangerous  materials,  forcible  entry  and  placing  water  on  a  fire)  that  can  be 
further  subdivided  into  individual  process  steps  (turning  a  meter  on,  placing  a  hose  line 
into  operation,  forcing  a  set  of  locks  on  a  door,  neutralizing  a  corrosive)  that  can  all  be 
measured  and  placed  on  a  scale  to  analyze  relative  performance.  Examples  of  certain 
measurable  inputs  and  outputs  are  numbers  of  operational  people  put  to  work  in  a  period 
of  time  for  an  operation.  The  number  of  victims  transported  and  whether  they  are 
ambulatory  or  non-ambulatory.  How  many  victims  were  triaged  at  the  scene  and  how 
many  medical  workers  were  needed  to  accomplish  this  effort? 

The  key  component  to  collecting  these  performance  measures  is  to  identify  those 
measures  that  will  actually  help  achieve  the  desired  goals  and  objectives  and  then  deliver 
those  metrics  to  the  right  people  at  the  right  time  (the  incident  commander,  the  chief  of 
training,  the  head  of  the  organization,  your  immediate  supervisor)  so  that  the  measures 
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can  be  analyzed  and  interpreted  so  an  action  can  be  taken  to  allow  the  proeess  to  eontinue 
or  a  defieiency  eorreeted. 

There  are  many  different  uses  for  metrics  and  performance  measures  ranging 
from  determining  present  performance  levels  to  predicting  future  ones  to  earefully 
eontrolling  an  existing  operation.  They  are  used  to  determine  the  following"^^: 

•  Baseline  -  so  that  we  know  where  we  are  beginning  or  where  we  are  at  the  present 
moment. 

•  Trending  -  shows  use  the  direetion  overtime  so  we  ean  monitor  the  operation  and 
ehange  proeedures  if  the  proeess  is  defleeting  from  the  ehosen  path. 

•  Control  -  to  determine  if  we  are  staying  within  a  predetermined  boundary  within 
our  proeess. 

•  Diagnostic  -  metries  and  performanee  measures  ean  help  identify  problem  areas 
in  the  proeess.  Areas  that  are  more  likely  to  eause  a  breakdown  of  the  system  or 
delay  the  outputs 

•  Planning  -  planning  performanee  measures  answers  the  question,  “given  eertain 
information  and  past  performance  levels,  what  ean  I  predict  and  plan  for  in  the 
future”.  This  is  the  most  important  type  of  measure  and  analysis  for  the 
emergency  responder  community.  This  type  of  performanee  measure  will  drive 
the  planning  and  training  eyele  and  thus  the  preparedness  level.  This  measure 
needs  to  get  to  the  right  people  at  the  right  time,  either  already  analyzed  with 
constructive  reeommendations  or  left  for  them  to  analyze.  These  will  be  the 
measures  that  ean  be  the  differenee  between  a  sueeessful  operation  or  ehaos  and 
injuries. 

4,  When  Do  You  Measure? 

Measurements  have  a  tendeney  to  measure  the  past  -  that  is,  the  things  that  have 
happened.  In  order  for  a  measure  to  be  of  value  to  the  organization,  it  must  transcend  the 
dimension  of  time  and  there  should  be  a  set  of  measure  for  eaeh  location  in  time,  the  past 
the  present  and  the  future.  Some  measurement  systems  may  very  well  be  able  to  measure 


Harbour,  Jerry  L.,  PhD,  The  Basics  of  Performance  Measurement,  Productivity  Press,  1997.  The  types  of  measures 
will  be  discussed  in  context  in  the  next  chapter. 
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in  all  three  loeations  in  time.  This  will  give  us  the  ability  to  know  where  we  were,  where 
we  are  at,  and  attempt  to  predict  where  we  will  be  in  the  future,  both  short  and  long  term. 

If  the  challenge  is  to  have  a  Squad^*^  company  ready  to  meet  a  certain  challenge  of 
protecting  an  special  event  that  will  take  place  at  a  future  date,  then  we  need  to  train  them 
to  take  on  the  responsibility,  exercise  them  to  make  sure  they  will  be  able  to  successfully 
complete  the  mission,  and  continue  to  challenge  them  so  we  can  predict  that  they  will  be 
continually  adapting  to  a  dynamic  climate.  Some  of  the  metrics  we  will  use  prior  to  the 
exercise,  during  the  exercise  and  following  the  exercise  may  look  like  this: 

•  Pre-exercise  -  examine  the  company  training  records;  examine  the 
competency  of  all  the  members,  and  the  dates  of  the  completion  of  these 
competencies.  This  will  give  us  a  baseline  on  where  we  are  before  entering 
into  the  performance  evaluation  process.  If  we  find  deficiencies  prior  to  the 
exercise,  then  we  will  be  able  to  correct  it  with  training  regiments.  In  order  to 
get  the  information,  we  may  just  use  a  simple  evaluation  form  asking  each 
member  where  they  think  they  are  in  relation  to  where  we  need  for  them  to  be. 
This  can  also  be  given  to  his  training  officer  in  his  unit  to  determine  his 
competency  on  a  specific  procedure  or  multiple  operations. 

•  Present  -  While  the  unit  is  running  through  an  exercise  or  an  activity, 
observers/evaluators  can  be  used  with  similar  check  sheet  or  logs  to  determine 
the  competency  of  the  unit.  The  important  thing  here  is  to  determine  the 
metrics  you  need  to  measure  to  give  you  a  clear  picture  of  the  units  ability  to 
be  prepared  for  this  special  event. 

•  Post  -  You  can  use  the  same  metrics  or  other  metrics  (have  the  company  fill 
out  evaluations  after  the  exercise  to  rate  their  performance,  have  an  after 
action  critique)  from  the  exercise  to  determine  the  direction  in  which  to 
proceed  to  accomplish  your  goal  of  having  the  company  ready  for  the  mission. 


A  Squad  Company  in  the  FDNY  is  a  multi-faceted  company  that  is  trained  in  all  aspects  of  the  fire 
service,  it  is  a  truck  company,  an  engine,  CFRD  trained,  Flazmat  Technician  unit,  a  light  rescue  company,  a 
decon  engine,  and  a  jack  of  all  trades. 
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5,  How  Do  You  Measure? 

Metrics  are  used  to  measure  the  knowledge  transfer  to  an  individual,  between 
team  members,  or  within  an  organization.  They  are  also  used  to  drive  performance  of 
individuals,  teams  and  organizations.  They  can  take  various  forms. 

Academic  measures  can  be  a  quantitative  test  with  a  measurable  outcome 
(example:  a  100  question  test  that  is  scored  on  a  percentage  right  basis).  Such  measures 
can  also  be  an  evaluation  process  -  questionnaires  that  evaluate  the  training  program,  the 
individual’s  knowledge  base,  the  instructor  or  leadership  of  a  group  or  the  knowledge  that 
was  transferred.  These  forms  of  metrics  bring  the  members  and  the  leadership  into  the 
process  and  give  them  a  stake  in  the  success  of  the  process.  These  measures  are  generally 
qualitative  but  can  easily  be  transformed  into  a  quantitative  scoring  methodology. 

Professional  measures  are  usually  qualitative  in  nature  and  does  not  affix  a 
ranking  to  a  member.  They  can  be  a  check-off  style  -  used  in  a  pass/fail  type  situation 
where  a  member  can  either  perform  a  competency  or  a  mission  essential  task  or  they  need 
to  receive  further  training.  Most  times  there  is  no  room  for  mediocrity  and  this  metric  can 
easily  be  quantified  by  tracking  the  number  of  members  correctly  performing  the 
evolution.  A  time  frame  can  be  added  to  record  relative  speed  to  complete  the 
competency.  They  can  be  a  questionnaire  style-  these  will  bring  the  members  directly  into 
the  measuring  process.  Questionnaires  have  the  most  dynamic  format  for  most  measures 
that  we  use.  They  can  ask  the  member  to  evaluate  his  knowledge  on  a  subject,  or  his 
attitude  about  a  team  or  the  leadership  of  the  team.  These  are  the  broadest  of  all  metrics 
but  can  be  used  to  receive  the  most  data  points  for  analysis. 

Whichever  metric  model  that  you  formulate,  you  need  to  make  it  specific  to  the 
exercise  that  is  running  and  must  be  able  to  measure  the  outcomes  that  the  stakeholders 
consider  most  critical.  They  must  also  be  simple  and  easy  to  grasp.  Remember,  if  your 
metrics  are  not  being  used,  then  maybe  you  are  measuring  the  wrong  thing. 

6.  Where  Do  You  Measure? 

Metrics  form  the  connection  that  will  drive  the  exercise  to  its  final  outcome.  They 
will  complete  the  loop  between  the  beginning,  or  baseline,  and  the  training  that  will  be 
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the  result  of  the  analysis  of  the  exereise.  All  phases  of  the  exercise  can  be  measured  and 
analyzed. 

The  pre-action  report  contains  all  the  information  on  the  present  capability  of  the 
members  participating  in  the  exercise  program.  During  this  pre-exercise  phase,  this  report 
will  form  the  baseline  of  what  shape  the  exercise  will  take,  an  evaluational  exercise,  a 
training  exercise  or  a  planning  exercise.  The  Exercise  Design  Team^'  will  design  an 
exercise  that  will  address  the  given  need.  All  pertinent  training  metrics  become  part  of 
the  evaluation  process.  SOP’s  and  procedures  will  form  the  goals  that  the  participating 
members  must  demonstrate. 

During  the  exercise,  measures  are  created  to  document  whether  individuals  or 
teams  meet  the  marks  or  the  minimum  standard.  These  measures  are  both  qualitative  and 
quantitative  and  are  recorded  and  formulated  into  an  After  Action  Report  (AAR). 
Recommendations  are  then  sent  to  the  stakeholders  (supervisors,  leaders,  commanders  or 
Fire  Chiefs)  as  actionable  items  that  can  be  used  to  direct  or  redirect  efforts  within  the 
organization. 

D.  THINGS  TO  REMEMBER  WHEN  DEVELOPING  METRICS 

In  order  to  be  successful  in  deploying  metrics  there  are  a  few  key  ideas  to 
remember. 

1,  Measurement  drives  behavior  A  metric  has  to  have  a  specific  use  to  a 
real  individual  or  organization  that  is  responsible  for  a  particular  outcome.  If  we  make  the 
metric  for  a  specific  person  then  he/she  takes  responsibility  for  that  measurement  and  will 
make  sure  it  gets  accomplished. 

2,  Measure  what’s  important  to  the  organization,  the  exercise,  the  unit 
performing  the  evolution  and  the  stakeholders.  Be  careful  not  to  allow  stakeholders  to 
exercise  too  much  influence  over  the  assessment  system,  because  they  may  (for  political 
reasons)  not  want  to  highlight  performance  gaps.  It  is  up  to  the  unit  members  and  their 
superiors  to  demand  the  best,  expect  the  best,  and  not  settle  for  anything  less. 


The  Exercise  Design  Team  is  described  in  the  DHS-Office  for  Domestic  Preparedness  publication,  Homeland 
Security  Exercise  and  Evaluation  Program,  available  at  www.ojp.usdoj.gov/odp,  March  2003. 

Harbour,  et  al.  p.  65 
Harbour,  et  al.  p.  67 
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3,  Include  comparative  basis  as  part  of  the  overall  program.  It  is 

important  build  metrics  that  are  consistent  from  one  exercise  to  another  in  order  to 
compare  and  measure  the  delta  gap  between  what  was  known  and  what  needs  to  be 
known.  Answer  the  question;  did  we  close  the  gap  and  get  where  we  need  to  be? 

4,  Metrics  should  be  collected,  distributed  and  analyzed  in  a  timely 
manner  for  actionable  items  and  targeted  to  specific  people  and/or  audiences.  Metrics 
are  the  product  of  an  exercise  that  closes  the  loop  on  the  process  from  beginning  to  end 
so  that  change,  if  needed,  can  be  accomplished.  A  series  of  metrics  and  exercises  can 
guide  the  path  to  real  institutional  change  if  needed  and  the  metrics  are  heeded. 

5,  Finally,  metrics  need  to  be  easily  understandable,  and  their  meaning 
needs  to  be  quickly  and  easily  grasped  and  understood.^'*  Time  is  valuable  and 
information  is  overwhelming.  The  analysis  needs  to  be  boiled  down  and  refined  before  it 
hits  the  target  audience.  Be  careful  not  to  “over  boil;”  that’s  when  you  run  the  risk  of 
biased  reporting.  It  only  needs  to  be  tender,  with  all  appropriate  information  from  the 
analysis  of  the  data  provided  by  the  metrics. 


Harbour  et  al. 
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V:  THE  PROPOSED  METRIC  SYSTEM 


A.  INTRODUCTION 

To  build  an  overall  proposal  for  consideration  by  the  leadership  of  FDNY,  this 
chapter  specifies  the  specific  metrics  system  developments  that  should  be  pursued  by  the 
organizations  that  need  to  help  the  leadership  restructure  the  exercise  system:  the 
Exercise  Design  Team  (EDT)  and  the  Executive  Committee  for  Preparedness  Evaluation 
(ECPE). 

B,  EDT  AND  ECPE  FUNCTIONS 

The  Exercise  Design  Team,  as  it  is  described  in  the  Department  of  Homeland 
Security’s  HSEEP  Program  (Homeland  Security  Exercise  and  Evaluation  plan)^^,  is 
essential  to  be  the  controlling  entity  to  bridge  the  gap  between  training  and  preparedness. 
The  Team  designs  the  exercise  to  evaluate  the  training  and  the  planning  process.  They 
become  the  independent  assessor  of  the  process,  and  the  reports  and  analysis  of  the 
metrics  that  they  generate,  prior  to,  during  and  post  exercise  are  presented  to  the 
leadership  and  the  stakeholders  of  the  organization.  This  analysis  forms  the  basis  for  the 
assessors  to  predict  the  preparedness  of  the  organization  to  respond  and  operate  at  a 
terrorist  event. 

The  Executive  Committee  for  Preparedness  Evaluation  is  comprised  of  an 
appropriate  group  of  qualified  personnel  who  are  knowledgeable  in  training,  operations, 
tactics,  exercises,  and  the  evaluation  process  should  act  as  the  main  assessors  of  the 
capability  and  preparedness  of  the  organization.  Examples  of  these  people  would  be  high 
ranking  officials  and  chief  officers  or  any  other  group  of  qualified  officers  that  will 
consistently  have  access  to  the  reports  and  act  upon  their  results.  They  would  act  as  the 
Executive  Committee  for  Preparedness  Evaluation  to  ensure  that  any  gap  that  is  identified 
through  this  process  is  acted  upon.  Whatever  form  this  committee  takes  for  the  individual 
municipality,  it  should,  as  a  basic  criteria,  be  comprised  of  the  most  knowledgeable  and 
qualified  members  of  the  organization  and  be  headed  by  a  staff  representative  that  has  the 


Homeland  Security  Exercise  and  Evaluation  Program,  Volume  1:  Overview  and  Doctrine,  U.S.  Department  of 
Homeland  Security,  Office  for  Domestic  Preparedness,  March  2003,  p.  23 
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power  and  ability  to  hold  entities  aeeountable  and  to  make  the  recommendations  happen 
that  will  close  the  gap  between  training  and  preparedness. 

The  Exercise  Design  Team  should  be  independent  of  Training  and  the  Executive 
committee,  but  should  issue  its  reports  and  analysis  to  both  entities.  This  would  ensure 
that  exercises  are  designed  to  test  and  evaluate  the  training  and  planning  process  and  limit 
biases  that  may  influence  the  true  analysis  process.  Any  weaknesses  or  strengths  will  be 
identified  and  recorded  so  that  the  truest  analysis  could  be  considered. 

The  rationale  for  the  creation  of  this  team  is  the  reduction  of  biases  that  could 
encroach  upon  the  system.  The  military  charges  the  commander  with  the  responsibility  of 
assessing  the  preparedness  and  readiness  of  the  command.  This  is  done  in  an  insulated 
environment  that  limits  opportunities  for  outside  influence.  By  using  multiple  experts  to 
balance  the  evaluation  process  the  EDT  can  accomplish  the  same  objective. 

C.  EXERCISE  DESIGN  CRITERIA  FOR  USE  BY  THE  EDT  AND  ECPE 

The  Exercise  Design  Team  should  keep  in  mind  a  number  of  suggestions  to 
ensure  success.  Eirst,  the  exercises  should  be  focused  on  definitive  goals  and  objectives. 
The  measures  that  will  be  used  should  be  simple,  measurable  and  specific.  More  is  not 
better. 

Second,  the  exercises  should  be  kept  simple  enough  to  control  and  measure  but 
not  so  simple  that  the  participants  will  be  insufficiently  challenged.  It  is  important  that  if 
you  have  members  participating,  they  should  be  challenged  by  the  tasks  for  which  they 
were  trained.  The  exercises  should  not  intentionally  set  them  up  for  failure  but  they 
should  not  be  guaranteed  success  without  adequate  performance.  There  are  lessons  to  be 
learned  in  failures  and  near  failures  as  well  as  successes.  The  exercise  should  be  used  to 
evaluate  but  also  reinforce  the  transfer  of  knowledge  through  the  learning  process. 

Third,  the  design  of  the  scenario  should  be  specific  and  based  in  reality.  The 
threat  level  of  a  particular  event  should  be  well  researched  and  topical. The  scenario 
that  is  used  should  be  thoroughly  thought  through  and  mapped  out  as  to  all  the  possible 
outcomes.  This  is  an  important  part  of  the  design  of  the  exercise  so  that  the  proper 

A  scenario  needs  to  use  agents  or  chemicals  as  they  would  react  if  released  at  the  given  time,  temperature  and 
conditions  that  are  prevailing  at  the  time  of  the  exercise  and  not  build  in  falsehoods  that  would  mislead  the  learning 
process  of  the  participants. 
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metrics  will  be  applied  for  analysis.  “Red  and  Blue  Teaming^^”  can  be  an  effective 
method  of  recording  all  possible  outcomes. 

Fourth,  the  measures  that  are  applied  should  be  built  into  each  exercise  and  can  be 
specific  for  that  exercise.  However,  some  of  them  should  also  be  adaptable  for  a  range  of 
different  exercises  so  that  a  common  measure  can  be  compared.  The  ideal  situation  is  to 
have  the  flexibility  to  “crosswalk”  through  different  exercises  and  compare  the  results  to 
draw  out  hidden  gaps  in  training.  There  are  several  different  skill  sets  and  knowledge 
bases  that  occur  within  the  fire  service  that  need  to  seamlessly  blend  together,  so  that 
fires  service  components  can  operate  as  a  team  to  complete  a  mission.  We  need  to 
measure  that  ability  to  operate  as  a  total,  interdependent  force  and  not  as  independent 
entities  that  have  the  potential  to  work  against  each  other. 

Fifth,  consequence  management  has  become  a  multi-jurisdictional  mission. 
Federal,  State,  and  local  assets  will  be  drawn  upon  to  control  the  incident  under  one 
Incident  Management  System.  The  main  emphasis  of  major,  multi-agency  exercises 
should  be  the  proper  evolution  of  Incident  Management  tasking,  and  the  expansion  or  de- 
escalation  of  an  Incident  Management  protocol.  This  is  a  key  element  to  the  management 
of  any  incident. 

D,  TYPES  OF  EXERCISES  TO  USE  WHEN  DESIGNING 

The  HSEEP  describes  seven  different  types  of  exercises  that  an  organization  can 
use  to  assess  their  training  and  evaluation  process  as  it  relates  to  their  assessment  of  their 
capabilities  and  preparedness  to  respond  to  a  terrorist  or  all-hazards  response  event.  They 
are: 

1.  Seminars  -  used  to  orient,  educate  and  update  participants  on  specific 
plans,  procedures  and  practices  within  an  organization.  It  is  generally  a  low  stress 
environment  comprising  of  discussions,  presentations  and  general  morale  building  for  an 
organization. 

2.  Workshops  -  used  to  develop  a  specific  product,  a  training  course, 
exercise  scenario  or  program.  They  are  generally  attended  by  specific,  skilled 

Red  and  Blue  teaming  is  a  technique  the  military  uses  to  test  offensive  and  defensive  strategies.  A  red  team  will  try 
to  foil  the  defensive  perimeters  and  the  blue  team  will  plan  for  the  defense  of  the  perimeter.  Both  plans  are  compared 
and  analyzed  to  improve  the  performance  of  both  offensive  and  defensive  entities. 
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representatives  (subjeet  matter  experts  or  other  professionals)  that  will  bring  a  skill  to  the 
table  in  ereating  the  produet. 

3.  Tabletop  Exercise  -  usually  attended  by  senior  staff  to  diseuss  and 
workout  key  issues  on  a  top  level  platform.  Tabletop  exereises  ean  test  polieies  and 
proeedures  or  the  planning  efforts  of  organizations  with  the  goal  of  improving  these 
produets  in  a  eontrolled  setting  and  identifying  possible  gaps  in  training,  operations  or 
planning.  Tabletops  are  a  useful  tool  for  mid-level  eommanders  and  ean  also  be  adapted 
to  lower  operatives  as  a  means  of  developing  leaders. 

4.  Games-  this  is  a  simulation  of  operations  that  ean  be  performed  on  a 
eomputer  based  platform  involving  one  or  more  individuals  or  teams.  Though  relatively 
new  to  the  fire  serviee,  the  true  potential  of  this  type  of  exereise  is  just  emerging. 

5.  Drills  -  these  are  eoordinated,  supervised  aetivities  that  tests  the 
eompeteney  of  speeifie  evolutions  or  skills,  for  single  members  or  teams  of  members. 
Drills  are  generally  foeused,  measured  against  established  standards,  eapable  of  having 
instant  feedbaek,  usually  performed  in  a  realistie  environment,  and  eondueted  in  a  way  so 
that  performanee  ean  be  isolated  and  easily  measured. 

6.  Functional  Exercises  -  designed  to  test  and  evaluate  individual  or  team 
eapabilities  in  the  eontext  of  a  realistie  environment  with  multiple  tasking  and 
interdependent  operations.  They  are  used  to  evaluate  performanee,  reinforee  established 
polieies  and  proeedures,  test  planning  efforts,  reinforee  training  objeetives  and  examine 
inter-team,  inter-ageney,  or  inter-jurisdietional  relationships. 

7.  Full-Scale  Exercises  -  large  seale  exereises  that  bring  all  parts  of  the 
training,  planning,  operational,  prevention  and  response  proeess  to  a  final  evaluation 
setting.  These  exereises  are  generally  used  to  assess  the  organizational  ability  of  players 
in  an  ageney,  inter-ageney  and/or  inter-jurisdietional  platform.  This  exereise  tests  both 
operational  as  well  as  non-operational  aspeets  of  a  eommunity’s  preparedness  eapability. 

Three  types  of  the  funetional  exereises  also  exist: 
a.  Training  Exercise 

This  exereise  is  in  the  eontext  of  a  training  eourse  for  the  purposes  of 
learning  a  speeifie  skill  set.  The  partieipants  will  be  told  what  they  are  learning.  Then, 
they  will  then  be  taught  what  they  need  to  learn,  they  will  apply  it  in  a  hands-on  exereise. 
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and  then  they  will  review  what  they  learned.  This  is  the  repetition  model  in  learning. 
They  will  be  measured  at  eaeh  step  and  observed  as  to  whether  they  did  the  evolution  or 
eompetency  eorreetly. 

b.  Evaluation  Exercise 

This  exereise  is  for  the  explieit  purpose  of  evaluating  performanee  of  an 
individual,  team,  unit  or  organization.  They  ean  be  a  “no  notiee”  exereise  where  they 
partieipants  will  be  brought  to  an  exereise  loeation  and  given  a  problem  to  solve  without 
any  warning.  The  exereise  is  fully  measured  and  graded  for  a  speeifie  skill  or  skills.  They 
ean  also  be  a  full  notice  exercise  where  they  individual,  team,  unit  or  organization  is  put 
through  an  exercise  that  they  had  full  knowledge  as  to  time,  place,  conditions  but  did  not 
have  a  complete  knowledge  of  the  scenario  that  would  be  presented  to  them.  This  type  of 
exercise  is  valuable  in  the  evaluation  of  the  whole  process  of  training  and  the  application 
of  transferred  knowledge. 

c.  Testing  Exercise 

This  type  of  exercise  is  a  full  notice  exercise,  usually  on  a  volunteer  basis 
to  test  a  plan,  procedure,  doctrine,  or  strategy.  The  participants  are  fully  briefed  before 
hand  and  the  exercised  is  fully  evaluated,  observed  and  measured,  not  on  participant 
performance  but  how  the  plan  or  procedure  affects  participant  performance. 

E.  MEASURES  TO  USE  IN  THE  EXERCISES 

The  metrics  that  can  be  used  and  embedded  as  part  of  the  exercise  should  be 
simple,  useful  and  focused  on  the  goals  and  objectives  of  the  exercise  (which  should 
mirror  the  strategic  plan  of  the  organization).  They  should  be  positioned  to  account  for 
conditions  pre-exercise,  during  the  exercise  and  post  exercise.  This  will  ensure  that  the 
past,  present  and  future  capabilities  are  being  measured  and  are  accounted  for  in  the 
analysis  of  the  data.  Each  set  of  metrics  should  be  specific  for  the  exercise  being  run, 
should  carry  a  common  theme  throughout  the  cycle  of  exercises  that  reflects  the 
benchmarks  of  the  strategic  planning  process. 

1.  Measures  for  Pre-exercise 

During  the  pre-exercise  phase  of  the  planning,  written  examination  results  or  final 
course  assessments  from  prior  training  courses  should  be  made  available  to  the  training 
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officer  of  the  unit  the  member  is  assigned  so  that  a  training  profile  can  be  developed. 
Every  member  should  have  a  training  portfolio  with  the  dates,  eourse,  instructors  and 
final  marks  recorded  along  with  evaluations  from  the  instruetors.  If  the  exereise  is  going 
to  be  evaluating  evolutions  and  eompetencies  that  are  not  normally  observed  by  the  unit, 
then  a  training  portfolio  should  be  sent  to  the  unit  well  in  advance.  This  should  be 
followed  up  by  individual  questionnaires  to  the  members  of  the  unit  that  will  be 
participating,  which  will  provide  data  to  quantify  their  self-assessment  of  their  training. 
This  will  be  repeated  in  the  post-exercise  phase  to  compare  results. 

This  gives  the  evaluator  of  the  exereise  a  baseline  framework  to  work  from  when 
objectively  evaluating  performance.  The  written  examination,  evaluation  of  training 
records  and  the  individual  questionnaires  will  give  the  assessor  a  feel  for  the  pre  exereise 
state  of  mind  that  the  participants  are  in  when  eomparing  to  the  end  result. 

2.  Measures  for  Use  During  the  Exercise 

The  evaluators  and  observers  should  be  prepared  with  an  evaluation  form  that  can 
be  used  in  a  eheck-off  or  format  or  permit  the  objeetive  recording  of  operations  during 
the  exereise.  Quantitative  measurement  of  specific  results  ean  also  be  used  as  a  measure 
of  time,  numbers,  counts  of  victims,  rescuers,  or  assets  used  or  staged. 

These  types  of  measures,  both,  objective  recording  by  evaluator/observers  or  the 
actual  counting  of  participants,  should  give  the  assessor  a  clear  picture  of  what  was 
happening  during  the  exereise  to  cause  a  speeific  chain  of  events.  All  of  these  factors 
should  be  present  in  the  final  report  of  the  analysis  of  these  data  points,  so  that  the 
assessors  ean  make  an  educated  measurement  of  the  participant’s  capability  and 
preparedness. 

3,  Measures  for  the  Post  Exercise 

Multiple  evaluations  might  be  used  in  the  post-exercise  phase.  A  repeat  of  the  self 
assessment  questionnaire  and  a  self  evaluation  by  the  participants  would  enable  them  to 
report  back  on  where  they  were,  and  what  they  did  or  didn’t  learn.  A  peer  evaluation 
questionnaire  could  examine  how  the  team  performed  during  the  exercise:  what  went 
right,  what  went  wrong  and  what  they  ean  do  better  in  the  future.  That  sort  of  assessment 
can  also  aid  efforts  to  improve  the  exereise  seenario.  A  leadership  evaluation  allows  the 
participants  to  rate  their  leaders  as  to  confidenee  in  their  abilities.  The  leader  evaluating 
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the  unit  can  produce  topical  discussion  points  that  can  improve  performance  for  both  the 
leadership  and  the  unit.  A  combination  of  these  assessments  can  give  the  evaluator  a 
sense  of  the  participant’s  perception  of  the  exercise,  a  sense  of  what  went  on  physically  in 
their  actions,  and  (more  important)  their  thinking  process  during  the  exercise.  The 
thinking  process  is  the  part  of  the  knowledge  gained  aspect  of  the  learning  process  that  is 
often  overlooked.  The  exchange  of  these  experiences  deepens  perpetuates  learning.  . 

These  measures,  together  with  the  evaluator’s  reaction  to  the  exercise,  become 
very  valuable  to  the  assessor  because  they  sum  up  the  evaluations  and  place  a  useful 
“control”  for  variations  between  groups,  evaluator/observers  and  the  participants.  There 
are  an  enormous  amount  of  data  points  that  can  be  plotted  from  these  questionnaires  or 
“reactionaries”  .  Submitting  all  these  measures  in  an  easy  to  read  report,  which 
compares  the  statistics  in  graphs  and  flow  charts,  can  help  the  Executive  Committee  for 
Preparedness  assess  the  capability  and  preparedness  of  the  uniformed  force. 

F.  ASSESSING  PREPAREDNESS:  DRAWING  ON  LEADERSHIP 

WISDOM 

The  fire  service  has  traditionally  blended  experience  and  knowledge  of  operations 
with  the  ability  to  perform  the  decision  making  process  in  a  high  stress  environment. 
Gary  Klein  in  “Sources  of  Power,”  (1998),  studied  the  ability  of  fire  commanders’  to 
quickly  and  effectively  make  the  correct  decision  while  being  under  the  high  stress  of  an 
operational  fire  or  emergency.  Klein  describes  this  as  the  Recognition-Primed  Decision 
Model.  Klein  describes  their  ability  to  fuse  two  processes  1)  the  size-up  of  a  situation, 
and  2)  their  recognition  of  the  proper  action  to  take  by  imagining  the  outcome  of  various 
options.  Fire  commanders  accomplish  this  fusion  in  seconds  by  pulling  in  their  career- 
long  experiences  and  accumulated  knowledge. 

This  process  is  what  will  make  our  ability  to  measure  their  performance  and  the 
ability  to  transfer  their  knowledge  to  less  experienced  and  future  chiefs  all  the  more 
difficult.  But  this  same  process  will  allow  us  to  exploit  the  ability  of  senior  experienced 
fire  chiefs  to  use  their  experience,  their  powers  of  observation  and  our  new  metrics  to 


57  A  phrase  coined  by  Industrial  Society,  2000,  in  the  report  by  the  IBS,  Kirkpatrick  and  beyond,  p3 1 . 
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effectively  assess  the  capability  and  preparedness  of  their  individuals,  their  units  and  their 
departments  to  respond  successfully  to  terrorism  events. 

To  bring  this  leadership  wisdom  to  bear  in  the  performance  evaluation  process, 
the  Fire  Service  should  mirror  the  military:  we  should  train  one  level  down  and  evaluate 
two  levels  up.  The  fire  service  should  mirror  this  philosophy  when  assessing 
preparedness.  Firefighters  should  be  trained  by  their  company  officers  and  their  training 
should  be  evaluated  by  battalion  and  deputy  chiefs.  Thus,  the  Exercise  Design  Team 
should  be  comprised  of  experienced  and  knowledgeable  officers  and  chiefs  who  should 
submit  their  analyzed  reports  to  a  committee  of  experienced  and  knowledgeable  chiefs  to 
assess  their  capability  and  preparedness  level.  To  aid  them  in  this  endeavor  I  propose  the 
following  system  for  assessment. 

G.  EVALUATION  SYSTEM  FOR  THE  FIRE  SERVICE 

1.  Purpose:  To  provide  a  logical,  simplified  system  for  commanders  and  the 
Executive  Committee  for  Preparedness  to  assess  in  concert  with  exercises,  the  capability 
and  preparedness  of  their  units  to  respond  to  terrorist  and/or  all  hazard  events. 

2.  Basic  Assumptions:  All  members  in  the  fire  organization  are  trained  to 
the  Hazardous  Materials  Operations  level  as  per  OSHA  1910.120  and  the  department 
adheres  to  NEPA  472.  The  skill  sets  for  all  hazard  response  for  operational  level  are 
complied  with  and  the  hazmat  team  meets  the  standards  of  Hazmat  Technician  as  per 
OSHA  1910.120. 

3.  Tools:  The  Executive  Committee  for  Preparedness  (ECP)  shall  be 
provided  with  the  following  reports  from  the  company  commander  of  the  participating 
units.  Reports  and  evaluations  on  the  capability  of  their  members  as  assessed  by  the 
training  officer  of  the  unit,  the  individual  training  portfolios  of  all  the  members,  the 
manpower  in  unit,  fully  manned,  percentage  of  non-assigned  members  working  tours  in 
unit/month,  percentage  of  time/month  a  covering  officer  is  working  in  unit,  and  reports 
and  evaluations  on  participants  in  exercises  as  representatives  of  their  individual  units. 

The  ECP  shall  be  provided  with  the  following  from  the  evaluators  and  observers 
of  the  exercise,  the  performance  evaluation  of  individual  members  and  the  performance 
evaluations  of  the  members  who  formed  teams  to  complete  tasks. 
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The  ECP  shall  be  provided  with  the  following  from  the  eommand  staff  of  the 
exereise,  the  performanee  evaluations  of  the  unit  in  eompleting  its  tasking,  the  ability  of 
the  unit  and  its  individuals  to  follow  orders  and  complete  their  tasking,  the  ability  of  the 
unit  to  coordinate  and  work  with  other  units,  and  the  ability  of  the  unit  to  coordinate  and 
work  with  other  agencies. 

Also,  the  ECP  shall  be  provided  with  a  quantitative  report  from  the  evaluators  on 
actual  numbers  of  measurements  used  as  compared  to  the  total  available  to  use.  (example; 
In  the  designed  scenario,  there  were  100  casualties  that  needed  to  be  decontaminated  but 
only  90  were  actually  decontaminated.  That  leaves  10  victims  that  were  unaccounted  for 
in  the  operation  that  could  be  spread  the  contamination  past  the  operational  borders.  Of 
course  the  units  do  not  know  how  many  victims  are  in  the  original  total.  This  measure 
points  to  a  gap  in  operations  which  needs  to  be  addressed). 

Einally,  the  experience  and  observations  of  the  Executive  Committee  for 
preparedness  shall  be  objectively  applied  to  evaluate  the  performance  of  the  units,  the 
Incident  Command  and  the  overall  operations  and  Command  and  Control  coordination  of 
the  participants. 

H.  STEPS  TO  FOLLOW  WHEN  ASSESSING  THE  RESULTS  OF  AN 

EXERCISE 

Step  1:  The  Exercise  Design  Team  (EDT),  working  independent  of  Training  and 
Operations,  most  likely  reporting  to  the  Planning  section  Chief,  will  establish  the  cycle  of 
exercises  using  tabletops  and  functional  exercises  as  their  main  performance  evaluation 
platforms.  The  EDT  will  use  the  standard  exercise  development  methodology  as  per  the 
National  Eire  Academy  and  the  Department  of  Homeland  Security’s  HSEEP.  They  will 
develop  and  embed  appropriate  performance  measurements  pre,  during  and  post  exercise. 

Step  2:  The  Executive  Committee  for  Preparedness  (ECP)  will  meet  and  become 
familiar  with  the  exercise,  its  goals  and  objectives,  and  its  support  of  the  strategic 
planning  process  of  the  organization. 

Step  3:  The  EDT  will,  upon  completion  of  the  exercise,  coordinate  and  submit  to 
the  ECP  a  complete  report  of  all  measures,  pre,  during  and  post  exercise  along  with  the 
completed  After  Action  Report  (AAR)  for  the  exercise. 
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Step  4:  The  ECP  shall  use  the  reports,  evaluations,  knowledge  of  the  units  and 
their  training  eapaeity  and  their  observational  experienee  in  observing  the  exereise  to 
establish  the  unit’s  eapability  and  preparedness  level  for  eaeh  of  the  evaluation  areas  used 
in  the  exereise.  The  ECP  shall  also  make  reeommendations  and  file  a  full  report  to  the 
Head  of  Training  and  the  Chief  of  Department  on  the  outeome  of  the  exereise  and  any 
training  or  operational  gaps  that  they  have  identified.  The  stature  of  the  ECP  shall  be  of 
sueh  nature  that  their  reeommendations  shall  be  held  in  the  highest  esteem  and 
implemented. 

Step  5:  The  ECP  shall  plaee  a  weighted  measure  upon  all  the  evaluations  similar 
to  the  following: 

1.  Evaluation  Performance  Measure  Index  (EPMI) 

The  Evaluation  Performanee  Measure  is  an  over  arehing  total  that  is  used  to 
determine  the  relative  performanee  of  an  entity  being  evaluated.  The  Index  is  heavily 
weighted  with  performanee  measures  that  are  used  during  exereises  but  also  evaluate 
faetors  that  will  infiuenee  the  performanee. 


Evaluation  Performance  Measure  Index  fEPMI) _ Points 


Manpower  5 

Training  5 

Equipment  &  equipment  knowledge  base  5 

Eeadership  aeeountability  5 

Pre-exereise  evaluation  5 

Exereise  performanee  measures  40 

Post  exereise  reaetionaries  5 

After  Aetion  Report  30 


Total  points  possible  100  points 

*Note:  Each  individual  organization  must  establish  the  priority  and  point  value  of 
each  of  the  measures  and  add  or  subtract  measures  in  accordance  to  the  strategy  of 
the  individual  organizations. 
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Once  the  weighted  scores  for  each  measure  is  deemed  appropriate  hy  the 
ECP,  the  point  totals  are  summed  and  a  relative  score  can  he  assigned  to  each 
performing  entity  and  skill  set.  This  is  just  a  relative  assignment  of  points  used  to 
compare  performance  between  units,  agencies  and  exercises.  The  weighted  values 
and  evaluation  points  will  change  as  deemed  appropriate  hy  the  ECP. 

Once  the  total  score  is  assigned,  the  ECP  can  group  the  units  in  a  level  as  depicted  in 
the  following  chart  or  plot  them  on  a  performance  line. 

2.  Relative  Preparedness  Index  (RPI) 


Relative  Preparedness  Index  fRPI)  Levels: 

Level  1  -  90-100  points  Fully  outfitted  and  trained  unit  ready  to  respond 

Fully  outfitted  hut  10%  of  unit  is  deficient  in  training  or 
untrained,  unit  capable  of  responding  and  operating 
Unit  is  lacking  some  equipment  or  20%  of  the  unit  is 
deficient  or  untrained.  Unit  can  respond  and  operate 
with  caution. 

Unit  is  lacking  equipment  or  has  50%  of  unit  deficient 
or  untrained,  unit  can  respond  and  operate  but  should 
be  teamed  with  a  fully  trained  unit. 

Unit  is  ill-equipped  an  ill-trained,  unit  can  respond 
under  the  extreme  caution  of  a  trained  unit  and  the  IC. 


Level  2  -  80-90  points 
Level  3  -  65-80  points 

Level  4  -  50-65  points 


Level  5  -  0-50  points 


Caution  must  be  used  when  establishing  and  interpreting  the  preparedness  levels. 
These  preparedness  levels  are  relative  and  relate  only  to  speeific  skill  areas  sueh  as  fire 
operational  response,  heavy  rescue  response,  or  hazmat.  A  distinction  must  also  be  drawn 
between  preparedness  and  readiness.  Units  or  agencies  may  profess  to  be  prepared  to 
respond  to  all  hazards  but  lack  the  capability  on  a  daily  basis  because  of  manpower, 
trained  personnel,  or  equipment  issues,  to  name  a  few  factors,  to  be  in  a  state  of 
readiness.  Capability  does  not  translate  into  preparedness,  and  preparedness  does  not 
translate  into  readiness.  The  evaluation  performance  measures  (EPM)  and  the  relative 
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preparedness  index  (RPI)  attempt  to  faetor  in  a  few  important  measures  that  will  allow 
the  incident  commander  to  best  utilize  the  assets  presented  to  him  at  an  operation. 

For  example:  a  Squad  company  of  the  FDNY  is  a  highly  trained  entity  in 
numerous  skill  areas.  These  areas  include  but  are  not  limited  to  fire  operations  (tenement, 
hi-rise,  commercial,  brush,  industrial,  private  residential,  multiple  dwellings,  vacant 
buildings,  row-frame.  Old  law  tenements,  new  law  tenements)^^,  confined  space 
operations,  collapse  operations,  water  rescue  and  scuba  diving,  hi-angle  rope  rescue, 
medical  emergencies,  and  hazmat.  Some  squads  may  be  more  proficient  at  certain  areas 
because  of  their  experience  in  response  to  these  areas.  Normal  Ladder  companies  may  be 
more  proficient  in  Hi-rise  fire  operations  with  a  level  1  preparedness  but  lack  hazmat 
capability  so  they  become  a  level  5  response  units  for  hazmat  skills.  This  knowledge  can 
be  of  extreme  importance  to  incident  commanders  while  they  are  developing  their 
strategies  at  incidents.  Though  all  chief  officers  have  a  general  idea  of  unit  assets,  those 
assets  change  from  tour  to  tour  and  week  to  week.  The  measures  within  the  relative 
performance  measures  attempt  to  reflect  those  factors  that  have  the  most  impact  upon 
preparedness  and  readiness. 

3.  Sample  Calculations 

Sample  calculations  to  use  when  figuring  out  weighted  points  for  specific  measures. 
Manpower  in  the  unit  calculation  -  weight  =10  points 
Assumptions:  5members  per  tour  working  in  unit 
5  members/tour  x  2  tours/day  x  30  days/month  =  300  man-tours/month 
#  of  tours  a  non-assigned  member  is  working  in  unit  =  points 


These  are  types  of  oeeupaneies  that  are  in  NYC  and  responded  to  by  FDNY.  Eaeh  oeeupaney  will  have 
different  taeties  and  proeedures  assoeiated  with  it  when  battling  a  fire  in  the  oeeupaney. 
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Assigned  member/tour 

man- tours/ month 

=  weighted  score 

5 

300 

= 

5 

4-5 

240  -  300 

= 

4 

3-4 

180-240 

= 

3 

2-3 

120-180 

= 

2 

1-2 

60-120 

= 

1 

0-1 

0-60 

= 

0 

-  The  ECP  shall  assign  the  weighted  score  depending  upon  the  units  manpower 
needs.  It  is  assumed  that  a  unit  will  operate  effectively  as  a  team  with  a  full  assigned 
roster  working  hut  will  degrade  as  a  function  of  outside  memhers  working  in  unit, 
-For  this  exercise  the  weighted  score  for  manpower  will  he  10  points  out  of  a  total  of 
100  points  available  on  the  performance  line.  This  weighted  average  will  change  for 
each  exercise  according  to  the  recommendation  of  the  ECP, 

-  Not  all  the  weighted  formulas  are  shown.  Units  score  could  vary  from  tour  to  tour 
and  it  would  he  a  very  useful  measure  for  the  Incident  commander  to  know.  All 
actions  to  either  use  a  particular  company  or  not  would  he  up  to  the  IC  and  his 
knowledge  and  understanding  of  the  system. 


Training  weighted/unit  calculation  =  10  points 

Assumptions:  Training  points  are  assigned  for  speeifie  training  areas  that  the  units 
are  responsible.  Squads  eompanies  are  responsible  for  the  most  tasks,  while  an  engine 
and  truek  eompany  are  only  responsible  for  fire  operations,  emergeneies  and  medieal 
responses.  The  ECP  ean  seleet  not  to  use  this  measure  for  eertain  units. 

#  members  x  hours  of  available  training  =  Total  hrs  of  unit  training 
Speeialized  Rescue  training  available: 


Rescue  technician 
Confined  Space 
Structural  collapse 
High  angle  rope  rescue 

Total  available  hours 


40  hours 
40  hours 
40  hours 
40  hours 

160  hours  X  #  members  assigned  to  unit  =  Points 
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160  hours  X  25  members  (Squad  Company)  =  4000  hrs. 
(4000  hrs  represents  a  fully  trained  Squad  Company  in  rescue  training) 


Points  available: 
Hours  of  training/unit 
4000  hours 
3000  -  4000  hrs. 

2000  -  3000  hrs. 

1000 -2000  hrs. 

0-  1000  hrs. 


#  of  points  assigned 

5  (represents  a  fully  capable,  prepared  &  ready  Squad) 

4  (may  represent  new  members  not  fully  trained) 

3  (may  represent  a  depletion  of  manpower  due  to  injuries, 
transfers,  etc.) 

2  (may  represent  serious  problems  in  manning  and  training) 
1  (may  represent  a  need  to  place  this  squad  out  of  service 
for  squad  responses,  this  becomes  a  decision  of  the  ECP)) 


Equipment  and  equipment  knowledge  base  can  be  graded  on  whether  the  equipment  is 
available  and  calibrated  and  whether  there  are  a  sufficient  number  of  members  that  are 
competent  in  the  use  of  equipment.  These  numbers  can  be  gathered  from  company 
records  of  equipment  logs  and  competency  logs. 

Leadership  calculations  are  based  upon  a  fully  staffed,  regularly  assigned  officers  to  the 
unit  that  are  fully  trained  and  competent,  and  that  posses  the  full  confidence  of  their 
members.  The  ECP  shall  assign  a  relative  number  according  to  the  above  criteria. 
Pre-exercise  evaluation  is  a  questionnaire  of  all  the  participants  that  establishes  their 
perception  of  their  capability  and  performance  in  operations,  also  hi-lighting  their 
confidence  in  their  leadership,  both  unit  and  command.  The  ECP  shall  generate  this 
number  from  the  questionnaires  as  compared. 
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Exercise  performance  measures  should  be  the  heaviest  weighted  measure  in  that  it 
measures  the  aetual  performanee  of  members  and  units.  The  number  is  gathered  from  the 
evaluation  reports  and  eheeklists  of  aeeomplishments  developed  for  eaeh  exereise.  The 
EDT  shall  gather  these  measures  and  report  them  to  the  ECP  for  final  adjustment  and 
analysis. 

Post  exercise  reactionaire  is  similar  to  the  pre-exereise  questionnaire  and  is  eompleted 
by  all  partieipants  whieh  is  then  analyzed  and  reported  to  the  ECP. 

After  Action  Report  is  another  heavily  weighted  measure  that  takes  into  aeeount  the 
observations  of  the  evaluators,  the  benehmarks  aehieved  by  the  units  and  their  operations 
planning  and  implementation. 

Total  Points  is  the  output  of  the  metrie  proeess  for  a  speeifie  exereise  or  aetivity  that  ean 
be  used  to  eompare  the  performanee  of  a  partieular  unit,  ageney  or  of  a  total  response. 
This  gives  the  ECP  a  measure  upon  whieh  to  assess  the  preparedness  of  the  organizations 
to  respond  to  the  exereised  hazard. 

Exereises  are  meant  to  be  vehieles  of  learning  and  evaluation  so  that  partieipants 
ean  experienee  new  operations  so  that  they  ean  reeall  and  apply  them  when  the  real 
ineident  oeeurs.  The  Eire  Serviee  must  use  exereises  to  inerease  eompeteney, 
preparedness  and  readiness  and  not  lose  the  opportunity  to  learn.  They  must  rejeet  the 
urge  to  eompete  and  not  be  eoneeited.  Terrorist  ineidents  require  a  eompetent,  prepared 
workforee  that  has  multiple  diseiplines  represented  that  ean  be  deployed  in  a  joint  foree 
to  attaek  the  problem  head  first  and  not  allow  it  to  beeome  a  one  ageney  show  that  is  a 
“jaek  of  all  trades  and  master  of  none”.  This  is  a  reeipe  for  failure  that  has  been  pointed 
out  by  the  medioere  exereise  system  presently  employed. 
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VI:  CONCLUSIONS  AND  RECOMMENDATIONS 


A,  INTRODUCTION 

An  important  opportunity  exists  to  improve  the  value  of  exercises  in  building  Fire 
Service  capabilities.  This  thesis  has  proposed  a  Metrics  Program  that  can  help  accomplish 
that  goal.  The  proposal  reflects  an  analysis  of  the  shortfalls  that  exist  in  the  current 
exercise  system,  and  builds  recommendations  on  how  to  fix  them  by  drawing  on  best 
practices  from  the  military,  private  sector  and  -  equally  valuable  -FDNY  itself  The  NEP 
has  given  us  the  template  to  help  guide  the  restructuring  of  the  exercise  system.  The 
military  has  shown  us  that  accountability  and  evaluation  by  superiors  is  a  hallmark  that 
will  drive  the  direction  of  training.  The  civilian  performance  models  have  showed  us 
why,  when  and  how  the  Fire  Service  can  better  measure  performance  as  part  of  a  broader 
system  to  make  the  Fire  Service  stronger. 

The  Exercise  Design  Team  (EDT)  and  the  Executive  Committee  for  Preparedness 
Evaluation  (ECPE)  will  play  a  key  role  in  this  process,  as  called  for  by  the  Homeland 
Security  Exercise  and  Evaluation  Program  (HSEEP).  This  thesis  specifies  planning 
guidelines  that  these  organizations  should  follow  in  developing  and  applying  metrics. 
Exercises  should  be  focused  on  goals  and  objectives  and  support  the  strategic  plan  of  the 
organization.  Exercises  should  be  simple  to  control  but  challenging  and  meaningful  for 
the  participants.  The  scenarios  should  be  realistic  and  the  measures  for  each  exercise 
should  be  specific  but  promote  comparisons  among  exercises  and  agencies.  Einally, 
exercises  that  are  multi-jurisdiction  or  multi-agency  should  stress  the  need  to  follow  an 
appropriate  Incident  Management  System  (IMS)  and  the  deficiencies  should  be 
emphasized  and  the  agency  should  be  held  accountable  for  the  deficiency  so  that  it 
becomes  correctable.  The  goal  is  not  to  produce  embarrassment,  but  rather  competency 
and  preparedness  that  can  be  objectively  measures  (and  improved,  when  deficiencies  are 
identified). 

B.  RECOMMENDATIONS 

1 .  EDNY  needs  to  develop  performance  metrics  that  provide  a  reliable  indicator 
that  the  training  and  exercise  system  has  successfully  achieved  knowledge  continuity  and 
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growth  within  the  organization.  Many  researchers  have  studied  the  way  that  organizations 
and  their  members  learn,  the  one  thing  these  scholars  all  agree  upon  is  that  in  order  to 
know  where  we  must  go,  we  must  remember  where  we  came  from  and  where  we  are  at 
the  present  time  so  that  we  can  assess  (and  improve)  our  performance  in  the  future.  Each 
step  along  the  way  requires  a  metric  or  set  of  metrics  to  guide  us  along  the  path. 
Moreover,  every  organization  should  have  a  strategic  plan  with  goals  and  objectives 
clearly  discemable,  and  the  training  that  an  organization  develops  must  lead  to  the 
achievement  of  the  organizations  objectives  and  goals.  The  only  way  that  an  organization 
will  know  if  it  has  met  its  goals  and  adhered  to  its  strategic  plan  is  through  the  metrics 
that  it  creates  to  measure  its  past,  present  and  future  performance  capabilities. 

2.  The  Fire  Service  has  always  trained  its  members  under  the  most  realistic 
conditions  that  it  could  reproduce.  This  is  reinforced  by  the  Army’s  practice  of  ‘train  as 
you  fight’  with  realism  in  both  equipment  and  environment.  What  the  Fire  Service  lacks 
is  creating  realistic  props  in  exercises  that  will  challenge  its  members  to  use  their 
experience  as  well  as  their  skills  to  solve  problems.  This  is  especially  true  for  homeland 
security-related  missions  (in  which,  I  pray,  few  live  events  will  provide  for  training 
experience).  Doing  this  will  present  an  air  of  urgency  and  danger  but  will  allow  more 
valuable  performance  evaluations.  Exercises  tend  to  be  ‘dumbed  down’  to  accommodate 
other  agencies  and  time  restraints.  FDNY  exercises  should  be  realistic  and  challenging  to 
all  participants.  This  will  translate  into  and  help  facilitate  accurate  performance 
measurements. 

3.  Else  the  measures  that  will  measure  only  the  aspects  of  an  exercise  that  are 
pertinent  to  the  goals  and  objectives  of  that  particular  exercise.  These  goals  and 
objectives  should  reflect  the  strategic  plan  of  the  organization.  More  measures  are  not 
better  -  and  too  many  are  an  impediment  to  the  effective  use  of  exercises  as  a  tool  for 
performance  assessment  and  improvement.  FDNY  exercises  should  only  measure  what 
we  will  use  and  implement. 

4.  Feedback  from  every  participant  should  be  encouraged.  The  feedback  should 
be  encouraged  prior  to  the  exercise,  at  appropriate  places  during  the  exercise  and  at  the 
end  of  the  exercise.  It  should  also  be  accepted  for  some  time  after  the  exercise  has  ended, 
after  participants  have  put  some  thought  put  into  their  experience.  The  military  does  a 
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good  job  of  accepting  feedback  from  its  soldiers  as  well  as  the  commanders.  Feedback 
can  and  should  be  used  by  the  FDNY  exercise  system  as  a  mechanism  to  produce  ehange. 

5.  In  the  military,  the  commander  is  responsible  for  assessing  the  eapabilities  of 
his  command,  supported  by  assessments  made  by  his  training  team.  The  Fire  service 
needs  to  adopt  the  praetice  of  assessing  capabilities  on  the  unit  level  as  well  as  the 
Commander  or  Chief  level.  The  military  has  worked  hard  to  isolate  its  training  proeess 
from  bias  and  stakeholder  influence.  To  aceomplish  the  same  objective,  the  Fire  Service 
needs  to  adopt  a  panel  of  members,  with  appropriate  rank,  training  and  experience  to  help 
provide  for  unbiased  assessments  of  performance. 

6.  Responsibility  for  training  is  a  personal  as  well  as  a  command  responsibility. 
There  must  be  aecountability  in  the  ranks  to  produee  a  competent  workforee  that  ean 
adapt  and  respond  to  any  operational  cireumstance.  Measurement  produees  a  pride  that 
will  inerease  performance.  The  Fire  Service  should  adopt  the  military  praetice  of  training 
one  level  down  and  evaluating  two  levels  up.  This  produces  accountability  up  and  down 
the  ranks  that  results  in  a  more  eohesive,  competent  work  force.  The  members  know  who 
they  are  aceountable  to  and  the  commanders  know  what  they  are  accountable  for  within 
their  command. 

7.  Training  standards  must  be  produeed  and  followed.  The  military  has  training 
standards  for  the  Force  that  they  adhere  to  when  training.  Presently  there  are  existing 
standards  for  hazardous  materials  response  and  mitigation  that  need  to  be  reinforced  by 
all  agency  participants  so  that  skill  sets  can  be  eompared  and  evaluated.  The  Fire  Service 
must  support  the  requirement  to  establish  standards  for  all  training. 

8.  After  Action  Reviews  (AAR)  must  beeome  the  standard  after  the  completion 
of  exercises  so  that  good  practices  ean  be  reinforced  and  deficieneies  identified  and 
corrected.  Deficiencies  should  be  addressed  immediately.  The  AAR  should  be  focused  on 
key  competeneies  derived  from  the  training  objeetives.  Emphasis  should  be  placed  on 
the  eorrect  proeedure  to  be  followed,  rather  than  pronouncing  judgment  on  suecess  or 
failure.  Leading  questions  should  be  asked  to  encourage  participants  to  self-discover 
important  lessons  from  the  exereise,  and  the  AAR  should  be  open  to  everyone  who 
participates. 
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9.  Stakeholders  in  the  event  should  be  identified  but  not  given  undue  influenee 
over  an  exereise.  Exercises  have  the  tendency  to  be  tailored  to  certain  organizations  to 
bolster  their  apparent  performance.  By  adopting  the  approach  to  metrics  proposed  in  this 
thesis,  the  risks  of  stakeholder  interference  can  be  minimized. 

C.  TOPICS  FOR  FUTURE  RESEARCH 

Improvements  can  always  be  made  on  evaluation  processes.  This  thesis  has 
outlined  multiple  performance  evaluation  models  that  exist  in  the  private  sector.  They  can 
be  used  in  multiple  ways  to  measure  preparedness  in  the  Fire  Service.  As  the  Fire  Service 
becomes  increasingly  specialized,  newer  and  better  training  methodologies  will  be 
explored.  Metrics  will  be  needed  to  measure  their  effectiveness  in  improving  competency 
and  knowledge  transfer  and  accumulation.  Similar  methodologies  need  to  be 
implemented  for  the  other  disciplines  that  respond  to  the  consequences  of  a  terrorist 
activity  so  that  they  can  be  coordinated  and  combined  under  the  Incident  Management 
System  (including  Faw  Enforcement,  Medical  and  Federal  response  agencies). 
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Table  2:  Comparison  of  different  models  of  training  evaluation^” 


Kirkpatrick 

Hamblin 

OEM  Model 

University 

IS  Carousel 

Phillips,  ROI 

KPMT 

GIRO 

Activity 

Accounting 

Identity  Business 
need;  define 
Development 
Objective;  design 
Learning  process 

Staged  process 
to  examine  the 
business  needs; 
design  solutions 
and  get  buy-in 

Context  analysis 

Input 

Reaction 

Reaction 

Input  Process 

Reaction 

Experience  the 
Learning  process 

Reaction  and 
planned  action 

Reaction  to 
training  and 
development 

Reaction 

Learning 

Learning 

Micro-acquis  tion 

Learning 

Use  and 

Reinforce  the 
learning 

Learning 

Learning 

Outcome 

immediate 

Behavior 

Job  behavior 

Micro 

Performance 

Transfer  of 
Learning 

Job  application 

Transfer  to  the 

workplace/ 

behavior 

Outcome 

intermediate 

Results 

Organization 

Macro 

Business  impact 

Judge  the 

Benefits  to  the 
organization 

Business  results 

Bottom  line 
added  value 

Outcome 

Ultimate 

Ultimate  value 

Mega  Societal 
Outcomes 

Social  impact 

Return  on 

investment 

Tamkin,  P.,  Yamell,  J.,  Kerrin,  M.,  “Kirkpatrick  and  Beyond:  A  Review  of  Models  of  Training  Evaluation”,  lES,  report  #  392,  2002,  pl3 
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Table  3:  A  model  of  evaluation^' 


•  Learning  from  others 

•  From  doing 

•  Formal  learning 

•  Lone/distance 


•  Self-effieaey 

•  Motivation  to  learn 

•  Ability 

•  leaminv 


•  Attitudes 

•  Skills 

•  Knowledge 

•  Meta  skills 


•  Relevanee 

•  Job  involvement 

•  Autonomy 

•  Climate  and  sunnort 


•  Capability 

•  Interpersonal  skills 

•  Self-management 

•  Creativitv/initiative 


Tamkin,  P.,  Yamell,  J.,  Kerrin,  M.,  “Kirkpatriek  and  Beyond:  A  Review  of  Models  of  Training 
Evaluation”,  lES,  report  #  392,  2002,  p29 
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